DEV Community: Sunny Bhambhani

Using Firehose as a Lambda Log Destination

Sunny Bhambhani — Thu, 15 Oct 2026 07:43:05 +0000

Using Firehose as a Lambda Log Destination

The last logging option available in the Lambda configuration is Amazon Data Firehose.

This option becomes interesting when CloudWatch Logs is not the final destination and you want Lambda logs to keep moving through a delivery pipeline.

Instead of just storing logs in one place, Firehose is built to receive data, optionally transform it, and then deliver it to another system.

How Firehose works at a high level

The flow is fairly straightforward:

Ingest: choose where the incoming data comes from, such as Amazon Kinesis Data Streams, Amazon MSK, or Direct PUT
Transform: optionally process or enrich the data before delivery
Deliver: send the data to a destination such as Amazon S3, Splunk, Dynatrace, and others

That flexibility is the main reason Firehose is useful. It can sit between the source and the final storage or observability platform without needing you to build the delivery logic yourself.

For this example, I kept things simple and used Amazon S3 as the destination.

If you want to see the full list of supported destinations, AWS documents them here:

https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html

The Firehose stream

Before sending Lambda logs to Firehose, I created a delivery stream with the following setup:

Source: Direct PUT
Destination: Amazon S3
ARN: arn:aws:firehose:eu-central-1:ACCOUNT_ID:deliverystream/PUT-S3-QdzyI

Make sure the status is Active after it is created.

Destination settings

The S3 delivery stream was configured like this:

S3 bucket: s3-lambda-logs
S3 bucket error output prefix: fh_err_
S3 bucket prefix: fh_
S3 bucket and error output prefix time zone: UTC
New line delimiter: Enabled

This setup is useful because it separates normal output from delivery errors and keeps the objects written to S3 organized under a predictable prefix.

Where the logs land in S3

After the stream is configured and Lambda starts sending logs through Firehose, the delivered files appear in S3 under a path like this:

Amazon S3 -> Buckets -> s3-lambda-logs -> fh_2026/ -> 10/ -> 11/ -> 12/

That structure makes it easier to find logs by date and process them later with downstream tools.

A quick note about delivery log classes

If you send logs to Amazon S3 or Amazon Data Firehose, Lambda requires a log group with the delivery log class.

That is an important detail because this log class is designed for delivery, not for the full CloudWatch Logs experience. In other words, you trade some CloudWatch features for simpler forwarding.

For example, delivery log class does not support standard CloudWatch Logs features such as Logs Insights and Live Tail.

It is also worth remembering that log events are only kept in CloudWatch Logs for two days when this class is used. That retention period is fixed and cannot be changed.

Function

The function remains the same which we discussed in the first article.

def lambda_handler(event, context):
    print("Dummy Lambda output")
    return {
        "statusCode": 200,
        "body": "Dummy Lambda output",
    }


if __name__ == "__main__":
    print(lambda_handler({}, None))

In this example:

A new log group is created named dummy_lambda_logs_to_firehose
I also created an IAM role for the Lambda function. The trust relationship and permissions are defined below:
- 03_iam_role_trust_relationships.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringLike": {
                    "aws:SourceArn": "arn:aws:logs:eu-central-1:ACCOUNT_ID:*"
                }
            }
        }
    ]
}

03_iam_role_permissiones.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": "arn:aws:firehose:eu-central-1:ACCOUNT_ID:deliverystream/*"
        }
    ]
}

Notes:

Log delivery to Amazon Data Firehose requires using a log group with the delivery log class. This enables efficient routing, but it does not support standard CloudWatch Logs features such as Logs Insights and Live Tail.

Final thought

If CloudWatch Logs is where you troubleshoot, S3 is where you archive, then Firehose is the option you use when logs need to travel.

For a simple demo, using Firehose with S3 keeps the architecture easy to understand.

In a larger environment, the same pattern can be extended to feed analytics, security, or observability platforms without changing the Lambda function itself.

Feel free to add your thoughts, happy learning!

Moving Lambda Logs to a Centralized Location [S3]

Sunny Bhambhani — Thu, 15 Oct 2026 07:19:33 +0000

Moving Lambda Logs to a Centralized Location[S3]

AWS Lambda gives you logs out of the box, but once an application starts growing, the real question becomes: where should those logs go?

If you are only testing a function, CloudWatch Logs is usually enough. But if you want long-term storage, centralized analysis, or integration with downstream systems, AWS gives you a few different ways to send Lambda logs somewhere else.

The available options

Inside the Lambda console, you can go to:

Configuration -> Monitoring and operational tools -> Logging configuration -> Edit -> Log Destination

From there, you get three choices:

Amazon CloudWatch Logs: the default option
Amazon S3: a simple choice for long-term archiving
Amazon Data Firehose: useful when you want to stream logs in near real time to other destinations

Each option solves a different problem, so the right one depends on what you want to do with the logs after Lambda writes them.

Function

The function remains the same which we discussed in the first article.

def lambda_handler(event, context):
    print("Dummy Lambda output")
    return {
        "statusCode": 200,
        "body": "Dummy Lambda output",
    }


if __name__ == "__main__":
    print(lambda_handler({}, None))

A quick note about delivery log classes

If you send logs to Amazon S3 or Amazon Data Firehose, Lambda requires a log group with the delivery log class.

That is an important detail because this log class is designed for delivery, not for the full CloudWatch Logs experience. In other words, you trade some CloudWatch features for simpler forwarding.

For example, delivery log class does not support standard CloudWatch Logs features such as Logs Insights and Live Tail.

It is also worth remembering that log events are only kept in CloudWatch Logs for two days when this class is used. That retention period is fixed and cannot be changed.

In this example, I used Amazon S3

For this setup, I created a new log group using the delivery class and named it dummy_lambda_logs_to_s3.

Then I selected this bucket:

arn:aws:s3:::s3-lambda-logs

I also created an IAM role for the Lambda function. The trust relationship and permissions are defined in:

02_iam_role_trust_relationships.json

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.amazonaws.com"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringLike": {
                    "aws:SourceArn": "arn:aws:logs:eu-central-1:ACCOUNT_ID:*"
                }
            }
        }
    ]
}

02_iam_role_permissions.json

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::s3-lambda-logs/*",
                "arn:aws:s3:::s3-lambda-logs"
            ]
        }
    ]
}

After saving the configuration, Lambda starts delivering logs to the S3 bucket.

What the S3 path looks like

When you invoke the function again, the logs show up in the bucket under a path like this:

Amazon S3 -> Buckets -> s3-lambda-logs -> AWSLogs/ -> ACCOUNT_ID/ -> eu-central-1/ -> dummy_lambda_logs_to_s3/ -> 2026/ -> 10/ -> 11/ -> 12/

That layout is useful because it keeps the logs organized by account, region, log group, and time.

Why this matters

This is the part that often gets overlooked.

CloudWatch Logs is great for day-to-day debugging, but not every team wants to keep everything there forever. Sometimes you want a cheaper archive. Sometimes you want to move logs into a data lake. Sometimes you want to push them to another analytics pipeline.

That is where S3 and Firehose start making sense.

Use CloudWatch Logs when you want convenience and fast troubleshooting
Use S3 when you want simple long-term storage and archiving
Use Firehose when you want logs flowing into another system in near real time

Notes:

Log delivery to Amazon S3 requires using a log group with the delivery log class. This enables efficient routing, but it does not support standard CloudWatch Logs features such as Logs Insights and Live Tail.

Final thought

There is no single best destination for Lambda logs. The best option depends on what you are trying to do with them.

If you need quick operational visibility, keep them in CloudWatch Logs.
If you need long-term retention, move them to S3.
If you need streaming delivery to another platform, Firehose is the better fit.

For this example, S3 is a clean middle ground because it keeps the setup simple while still giving you centralized storage.

Feel free to share your thoughts, happy learning!

Lambda Logging in AWS [Default]

Sunny Bhambhani — Thu, 15 Oct 2026 07:01:03 +0000

Lambda Logging in AWS

This example uses a very small AWS Lambda function named dummy_output to show what Lambda logs look like by default, why plain-text logs are sometimes not enough, and why structured logging becomes important once you need centralized observability.

Lambda function

The function prints a dummy message and returns a simple success response.

Function

def lambda_handler(event, context):
    print("Dummy Lambda output")
    return {
        "statusCode": 200,
        "body": "Dummy Lambda output",
    }


if __name__ == "__main__":
    print(lambda_handler({}, None))

Response

{
  "statusCode": 200,
  "body": "Dummy Lambda output"
}

Default log output

When the function runs, AWS Lambda automatically sends logs to CloudWatch. By default, those logs are often plain text and include both application output and platform/runtime details.

START RequestId: d386d5a9-2a75-41d1-TRIMMED Version: $LATEST
Dummy Lambda output
END RequestId: d386d5a9-2a75-41d1-TRIMMED
REPORT RequestId: d386d5a9-2a75-41d1-TRIMMED
Duration: 2.40 ms   
Billed Duration: 110 ms 
Memory Size: 128 MB 
Max Memory Used: 38 MB  
Init Duration: 106.81 ms

By default, these logs usually appear in a CloudWatch log group such as /aws/lambda/FUNCTION_NAME, In our case its /aws/lambda/dummy_output.

Why plain text is limited

Plain text is fine for quick debugging and small experiments. The problem starts when you need to centralize logs across multiple Lambda functions, search them efficiently, correlate events, or build dashboards and alerts.

At that point, unstructured text becomes harder to query and harder to enrich. A log pipeline can still ingest plain text, but it will usually need extra parsing rules to make the data useful.

Why structured logs help

If your application is fully Lambda-based and you want to understand how it behaves in production, structured logs are a better default.

JSON logs make it easier to:

search by fields like Level, RequestId, Service, etc
filter application errors without parsing free-form text
send logs to a centralized platform
build dashboards, alerts, and retention policies
correlate application logs with platform and tracing data

Lambda logging configuration

AWS Lambda supports changing log formatting from the console:

Configuration -> Monitoring and operational tools -> Logging configuration -> Edit -> Log format -> JSON

A practical starting point is:

Application log level: INFO
System log level: WARN

That keeps routine noise down while still preserving useful runtime and platform information.

JSON log output

When JSON logging is enabled, the response from the function does not change, but the logs become much richer and easier to work with.

{
  "time": "2026-10-11T11:48:32.684Z",
  "type": "platform.initStart",
  "record": {
    "initializationType": "on-demand",
    "phase": "init",
    "runtimeVersion": "python:3.14.mainline.v75",
    "runtimeVersionArn": "arn:aws:lambda:eu-central-1::runtime:TRIMMED",
    "functionName": "dummy_output",
    "functionVersion": "$LATEST",
    "instanceId": "2026/10/11/dummy_output[$LATEST]TRIMMED",
    "instanceMaxMemory": 134217728
  }
}

You can also see the runtime start, execution, and final report details separately:

{
  "time": "2026-10-11T11:48:32.807Z",
  "type": "platform.report",
  "record": {
    "requestId": "5adc1477-4705-4710-TRIMMED",
    "metrics": {
      "durationMs": 1.914,
      "billedDurationMs": 102,
      "memorySizeMB": 128,
      "maxMemoryUsedMB": 38,
      "initDurationMs": 99.535
    },
    "status": "success"
  }
}

What this shows

This is not about doubting AWS Lambda itself. Lambda is already a managed serverless compute service, and AWS handles the platform for us.

The real reason to monitor it is different: our application logic is still ours. We still need to know when it fails, how it behaves under load?, what it returns?, and how it performs over time?.

If 100% or majority of our application runs on Lambda, centralized logging becomes essential for:

troubleshooting failures
tracing request paths across services
understanding latency and cold starts
detecting patterns in errors or retries
supporting audit and operational visibility

Notes:

The default CloudWatch Logs destination keeps the standard CloudWatch Logs experience, including Logs Insights and Live Tail. Query cost is separate, so broad or repeated searches can increase the total.

Conclusion

For quick experiments, plain text logs are acceptable.

For production, JSON logs are usually a better choice because they are easier to aggregate, query, and route into a centralized observability stack.

If the goal is to operate a Lambda-based application at scale, structured logging should be the default rather than an afterthought.

Feel free to share your thoughts, happy learning!

Python - Stop storing passwords in plain text! A guide to werkzeug.security

Sunny Bhambhani — Mon, 05 Jan 2026 18:27:12 +0000

Problem

Storing passwords in plain text presents numerous dangers.

What happens if your database is compromised?
What if you accidentally reveal the wrong window while demonstrating and sharing your screen?

Such mistakes can result in serious issues, including losing the trust of customers, damaging brand value, financial repercussions, legal penalties, etc. These kinds of incidents are not rare; news reports about data breaches frequently appear worldwide.

Refer to https://databreaches.net/ for more details around data breaches worldwide.

Therefore, we should consider security in each aspect of application development, starting from planning, coding, building, deployment, etc.

And in this article, we will be talking about hashing passwords, but how do we actually do that? What options are available, etc.?

There are numerous options available, out of which some of the famous ones are werkzeug.security, argon2, bcrypt, etc.
Here we will see what werkzeug.security is and its practical implementation.

Prerequisites

Working Python setup.
Werkzeug Library installed.

pip install werkzeug

Basics

How does this work?
werkzeug.security provides helper functions for secure password storage and verification, and the two most commonly used functions are:

generate_password_hash(password, method=XXX, salt_length=XXX)
check_password_hash(PASSWORD_HASH, ACTUAL_PASSWORD)

More details around this can be found in its official documentation, refer: https://werkzeug.palletsprojects.com/en/stable/utils/ for more details.

The basic principle behind it is,

generate_password_hash takes the actual plain-text password as an input.
Further generates a salt of specified length.
The password + salt is processed through the chosen algorithm.
- scrypt(default)
- pbkdf2
Finally, it generates the output as a structured, hashed string.
Then check_password_hash comes into the picture, it securely checks if the given password and the stored password hash match or not.
Once that is evaluated, it produces a Boolean result, either true or false.

HOWTO

generate_password_hash

# Import generate_password_hash
from werkzeug.security import generate_password_hash

# Plain text password, if using Flask, it can come from any web form
plain_password = "SHH_SECRET"

# Generate a hashed string using default parameters
hashed = generate_password_hash(plain_password)

# Printing a hashed string
print("Hashed String: ", hashed)

# Output
Hashed String:  scrypt:32768:8:1$A2qi9GPp8Ut3cESv$31e31ffc4d21251396b391c31f09f6637e9169d4c5541a790cf0d9fcae3d58d2ba8a2df51e13f2601ba77fabe2e68d517fe845c63b489418309791ae1e350d3b

If you observe, the hashed string contains multiple sections
scrypt:N:r:p(METHOD_and_PARAMETERS)$salt(RANDOM_SALT)$derived_hash(ACTUAL_HASHED_PASSWORD)

In the above example, we just used the default values to generate the password hash.

Method: scrypt
N: 32768
r: 8
p: 1

Quick note about the parameters: Based on my understanding, the parameters N, r, and p in scrypt directly control how much compute and memory are required to derive a single password hash. The larger the numbers, the harder and more computationally intensive it is to break the password.

If you want to use custom parameters, this is how we can use them:

generate_password_hash(plain_password, method="scrypt:32768:16:1", salt_length=32)

It will produce the output as:

scrypt:32768:16:1$0hkax1LolbhqTyEN9rMSi8MC2MoF5B8w$85a7175deab3795db17898d17f03e3164df0ae1d158081ca3ed1680f9f940ff9cac76df18a71595160fd39963f4b4811c82348354e0c4a67a328104c56900230

if you want to use another method, this is how we can use it:

generate_password_hash(plain_password, method="pbkdf2:sha256")

The execution of generate_password_hash function generates a string that needs to be stored and checked for verification.

check_password_hash

# Import generate_password_hash and check_password_hash
from werkzeug.security import generate_password_hash, check_password_hash

# Plain text password, if using Flask, it can come from any web form
plain_password = "SHH_SECRET"
# Generate a hashed string using default parameters
hashed = generate_password_hash(plain_password)

# Login attempt 1 with wrong password
# If using Flask, it can come from any web form, like a login screen
login_attempt_1 = "SHH_WRONG_SECRET"
is_valid = check_password_hash(hashed, login_attempt_1)
print("Password correct?", is_valid)

# Login attempt 2 with correct password
# If using Flask, it can come from any web form, like a login screen
login_attempt_2 = "SHH_SECRET"
is_valid = check_password_hash(hashed, login_attempt_2)
print("Password correct?", is_valid)

# Output
# This is the first attempt in which we used the wrong password, and it concluded that it was incorrect, hence it gave the output as false
Password correct? False
# This is the second attempt in which we used the correct password, and it concluded that it was correct, hence it gave the output as true
Password correct? True

Conclusion

Never store plaintext passwords; store only the hash strings.
Do not re-invent the wheel; we can easily rely on vetted libraries and generate hash strings.
Keep your dependencies up-to-date.
Don’t compare strings manually; we can easily leverage functions like check_password_hash, which handle secure comparison.

Soon, I will be adding details about other libraries and functions as well. I hope this will be helpful.

Feel free to add your thoughts and experiences. Also, since I am still learning, correct me if I am wrong or have misinterpreted anything. Happy Learning! 😄

A smarter way to use imagePullSecrets in Kubernetes Cluster using ServiceAccounts

Sunny Bhambhani — Mon, 12 May 2025 12:16:05 +0000

Kubernetes is everywhere now a days, so does the container images and fetching the images from a private registry is a norm because of N number of reasons including security, that being the topmost.

Recap

Just to give you a heads up before we dive further, let's assume I fire kubectl create deployment nginx --image nginx what do you think happens?

Using kubeconfig, kubectl first authenticates with the API server.
It then constructs a deployment object using the values which we provided like deployment name as nginx, image to be used as nginx and some default values and then sends a request to the API server.
API server then validates the request and the object definition.
This is further stored in etcd datastore.
Then the controller manager specifically deployment controller comes into the picture which creates a replica set.
Then the replica set creates a pod (this is where our nginx image will be used).
Then the schedular assigns the pod to a node, where it will run based on multiple conditions. If interested please check out my previous article on what all things happens in the background: Resource management in Kubernetes
Once the node is identified, the kubelet running on the node is responsible for the management of the pod.
Once the pod is scheduled on a specific node, the kubelet looks for the image to be used and if it is already present.

if that is present locally, it will use it.

If not, it pulls the image from the registry.

If the registry is public, there are no issues, it just pulls the image, runs the pods, and we are done.

When imagePullSecrets are used?

Assume the registry is private, this is where imagePullSecrets come into play.
Kubernetes will need the secrets to authenticate the registry, so that it can pull the image.

How it works?

Firstly, we need to create a secret which contains the credentials and some details of our docker registry like server-name, username, password, email, etc.
It can be created both imperative as well as declarative way.

$ k get secrets  -o yaml dockersec
apiVersion: v1
data:
  .dockerconfigjson: ENCODED JSON OBJECT
kind: Secret
metadata:
  name: dockersec
  namespace: default
type: kubernetes.io/dockerconfigjson

$ k create secret docker-registry dockersec --docker-server=DOCKER_REGISTRY --docker-username=USERNAME --docker-password=PASSWORD

Quick note: The secret must be of a type kubernetes.io/dockerconfigjson or docker-registry based on the way you choose to create the secret.
Once the secret is created, it must be attached to a pod so that it can use it.
Just add below content in your pod/deployment/statefulset yaml files, the name of the secret for me is dockersec but you can change based on your requirements.

spec:
  imagePullSecrets:
    - name: dockersec

Before pulling the image, the kubelet checks if the image is from a private registry. If so, it looks for imagePullSecrets in the spec.
Without proper imagePullSecrets, the image pull will fail with ImagePullBackOff.

Problem

Now assume you have hundreds of YAML files and hundreds of Kubernetes Object where we have to use this imagePullSecrets.
Just for the sake of doing once, you thought to do it manually but just imagine there was a change in the name of the Kubernetes secret.
Maybe because the organization now decides to streamline the naming convention of all the Kubernetes objects based on environments like prod/preprod/staging/etc.
- Or your org is now moving to a different registry or a different provider.
- Or the application is getting re-factored based on different namespaces or functions.
- There can be N number of reasons and doing all these changes again is a tricky and tiresome task.

One way to look at it is, use of helm charts which can help us to manage all the Kubernetes objects and this kind of a change can be pretty straight forward provided the helm charts are structured correctly and using values file we can just update the name of the secret and then can upgrade the chart, and we are done.

But if in case we are not using and package manager like helm and we are just working with vanilla YAML files then for that as well there is a neat way to do it.

The NEAT way!

We can attach the imagePullSecrets with the serviceaccounts and where-ever the serviceaccount is used, it will automatically populate the imagePullSecrets in the respective object like Pods.

Below is the current status of my Kubernetes objects, no secrets, a default serviceaccount and no imagePullSecrets updated in the pod.

$ k get secrets
No resources found in default namespace.
$ k get sa
NAME      SECRETS   AGE
default   0         33m
$ k get pods
NAME                     READY   STATUS             RESTARTS   AGE
nginx-65fd88fc88-8xjc9   0/1     ImagePullBackOff   0          32m

Create the docker-registry secret

$ k create secret docker-registry dockersec --docker-server=DOCKER_REGISTRY --docker-username=USERNAME --docker-password=PASSWORD
secret/dockersec created
$ k get secret
NAME        TYPE                             DATA   AGE
dockersec   kubernetes.io/dockerconfigjson   1      3s

Patch the default serviceaccount to have the imagePullSecrets. It can be any serviceaccount based on your requirements.

$ kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "dockersec"}]}'
serviceaccount/default patched
$ k get sa
NAME      SECRETS   AGE
default   0         35m

Or you can manually edit the serviceaccount as well and add below content.

$ k get sa default -o yaml
apiVersion: v1
imagePullSecrets:   # ADD
- name: dockersec   # ADD
kind: ServiceAccount
metadata:
  name: default
  namespace: default

To see this in action now, delete the pod which is in failed state.

$ k delete pod nginx-65fd88fc88-8xjc9 --force

Check the events of the pod, now it was successfully able to pull the image from the private registry.

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  65s                default-scheduler  Successfully assigned default/nginx-65fd88fc88-w8mph to minikube
  Normal   Pulling    64s                kubelet            Pulling image "sunnybhambhani/test:v1"
  Normal   Pulled     37s                kubelet            Successfully pulled image "sunnybhambhani/test:v1" in 30.288s (30.288s including waiting). Image size: 567185619 bytes.
  Normal   Created    21s (x3 over 36s)  kubelet            Created container: test
  Normal   Started    21s (x3 over 36s)  kubelet            Started container test

Interesting bit is imagePullSecrets are automatically added to the object.

$ k get pods nginx-65fd88fc88-w8mph -o yaml | grep -i serviceaccountname
serviceAccountName: default
$ k get pods nginx-65fd88fc88-w8mph -o yaml | grep -i imagepullse -A 1
  imagePullSecrets:
  - name: dockersec

Bonus Tip

In case the pod is not in running status, it can be due to multiple reasons.
Just describe the pod to see what is happening in the background.
kubectl describe pod POD_NAME | awk '/^Events:/ {found=1; next} found', it looks for the line starting with Events, once found prints the remaining lines.

$ kubectl describe pod nginx-65fd88fc88-8xjc9 | awk '/^Events:/ {found=1; next} found'
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  7m21s                   default-scheduler  Successfully assigned default/nginx-65fd88fc88-8xjc9 to minikube
  Normal   Pulling    4m17s (x5 over 7m20s)   kubelet            Pulling image "REGISTRY/test:v1"
  Warning  Failed     4m14s (x5 over 7m17s)   kubelet            Failed to pull image "REGISTRY/test:v1": Error response from daemon: pull access denied for REGISTRY/test, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
  Warning  Failed     4m14s (x5 over 7m17s)   kubelet            Error: ErrImagePull
  Warning  Failed     2m41s (x19 over 7m16s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    2m20s (x21 over 7m16s)  kubelet            Back-off pulling image "REGISTRY/test:v1"

Here the logs clearly state, requested access to the resource is denied and in-turn the status of the pod is ImagePullBackOff.

$ k get pods
NAME                     READY   STATUS             RESTARTS   AGE
nginx-65fd88fc88-8xjc9   0/1     ImagePullBackOff   0          76s

References:

Google.
Kubernetes docs (https://kubernetes.io/docs/home/).
kubectl explain.
Linux man pages.

PS: This works with any flavor of Kubernetes: minikube, Elastic Kubernetes Service(EKS), Azure Kubernetes Service(AKS), Google Kubernetes Engine(GKE), etc.

I hope this will be helpful, feel free to add your thoughts and experiences, Happy Learning! 😄

Kubernetes hardening made easy: Running CIS Benchmarks with kube-bench

Sunny Bhambhani — Mon, 03 Mar 2025 04:41:17 +0000

In today's world, where security risks and breaches are growing daily, it is crucial to maintain our applications and infrastructure's compliance with security standards and that is where CIS benchmarks from CIS (Center for Internet Security) comes in. And with kube-bench, running these checks becomes straightforward, helping you strengthen your Kubernetes clusters with confidence.

CIS Benchmarks

Here is the mission statement from CIS's website.

Our mission is to make the connected world a safer place by developing, validating, and promoting timely best practice solutions that help people, businesses, and governments protect themselves against pervasive cyber threats.

Ref. https://www.cisecurity.org/ for more details.

It basically helps us to jot down:

What all best practices/guidelines to use?
Which tools are available to assist us in scanning our application or infrastructure?
Making sure the best practices are updated on timely manner.

The benchmarks are basically available for multiple platforms including:

All the public cloud providers.
Softwares
DevSecOps tools
Mobile devices
Operating Systems
and much more.

You can find their extensive list here: https://learn.cisecurity.org/benchmarks

CIS Benchmarks: Download PDF for your platform

For downloading the PDF files you just need to provide some general details about yourself.

Once done, press "Submit" and you will receive an email with the link from where you can download all kinds of benchmarks.

It looks something like this:

Upon pressing "Access PDFs" you will be routed to their extensive list of benchmarks which looks something like this:

You can choose whatever you want to, but since we are interested in Kubernetes as of now, we will scroll down to the Kubernetes section.

And the good part here is, this is not limited to vanilla Kubernetes, we can even download benchmarks for all the Kubernetes flavours like EKS from AWS, AKS from Azure, GKE from Google, etc.

Once you download a specific PDF, you will see it contains a great amount of details including recommendations, problem statement, impact, remediation, etc.

Now, if you have observed the PDF contains thousands of recommendations and going thru them and applying them one by one is a time-consuming task and just imagine you have 100s of clusters. Though I would recommend you to at-least go thru it once and get an idea of what all details it contains and how we can make use of it.

To make our lives easier there are couple of tools which can help us to automate this process and help us to identify where we are lacking. So that we can fix them quickly.

CIS Benchmarks: Tools

There are couple of tools which are managed by CIS itself like CIS-CAT Lite/CIS-CAT Pro/etc. CIS-CAT Lite is a free version and it supports a limited options excluding Kubernetes. CIS-CAT Pro is the one which supports Kubernetes but it is just available for CIS SecureSuite Members.

Ref. https://www.cisecurity.org/cybersecurity-tools for more details about the tools.

Now, let's talk about the good part, the community has given us couple of opensource tools which does the same 😉 the most used one is kube-bench (from Aqua Security).

HOWTO: kube-bench

kube-bench is a tool that checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark.

There are multiple ways to run kube-bench

Install it via a package manager

Download the latest package based on your OS from github releases page and install it.
Ref: https://github.com/aquasecurity/kube-bench/releases
Since, I am using Ubuntu, I will use dpkg to get this package installed.

$ wget https://github.com/aquasecurity/kube-bench/releases/download/v0.10.2/kube-bench_0.10.2_linux_amd64.deb
$ sudo dpkg -i kube-bench_0.10.2_linux_amd64.deb
Selecting previously unselected package kube-bench.
(Reading database ... 41333 files and directories currently installed.)
Preparing to unpack kube-bench_0.10.2_linux_amd64.deb ...
Unpacking kube-bench (0.10.2) ...
Setting up kube-bench (0.10.2) ...
$ kube-bench version
0.10.2

Run as a job in the Kubernetes Cluster

Get the job.yaml from the same github link and deploy it on your cluster.
Ref: https://github.com/aquasecurity/kube-bench/blob/main/job.yaml
Based on your platforms there are multiple job*.yaml files available like for EKS, AKS, GKE, etc.

$ kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
job.batch/kube-bench created

$ k get pods
NAME               READY   STATUS      RESTARTS   AGE
kube-bench-n22k9   0/1     Completed   0          96s

$ k logs kube-bench-n22k9 | head -n 20
[INFO] 1 Control Plane Security Configuration
[INFO] 1.1 Control Plane Node Configuration Files
[PASS] 1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated)
[PASS] 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated)
[PASS] 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated)
[PASS] 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated)
[PASS] 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated)
[PASS] 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated)

OUTPUT TRIMMED

$ k logs kube-bench-n22k9 | tail -n 20
OUTPUT TRIMMED

5.7.3 Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a
suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker
Containers.

5.7.4 Ensure that namespaces are created to allow for appropriate segregation of Kubernetes
resources and that all new resources are created in a specific namespace.


== Summary policies ==
0 checks PASS
6 checks FAIL
29 checks WARN
0 checks INFO

== Summary total ==
57 checks PASS
19 checks FAIL
54 checks WARN
0 checks INFO

Just check the logs of the pod for all the recommendations. These logs contain multiple sections for different-different Kubernetes components like Controlplane, ETCD, Worker nodes, etc.
At the end of each section you can see statistics about the checks and at the end of the logs you will see a quick summary about the total (as mentioned above).

== Summary node ==
14 checks PASS
2 checks FAIL
8 checks WARN
0 checks INFO

Run as a docker container

docker run --rm --net host --pid host --user 0 \
  -v /etc:/etc:ro \
  -v /var:/var:ro \
  -v /usr/bin:/usr/bin:ro \
  -v /usr/lib:/usr/lib:ro \
  aquasec/kube-bench:latest --benchmark cis-1.24

You can change the version based on your requirement, 1.24 is the latest one.
More details about the versions can be found here: https://github.com/aquasecurity/kube-bench/tree/main/cfg
For this as well, just check the logs of the container you will find the same results as we described in earlier section.

This is how the remediation steps look like:

== Remediations master ==
1.1.9 Run the below command (based on the file location on your system) on the control plane node.
For example, chmod 600 <path/to/cni/files>

1.1.11 On the etcd server node, get the etcd data directory, passed as an argument --data-dir,
from the command 'ps -ef | grep etcd'.
Run the below command (based on the etcd data directory found above). For example,
chmod 700 /var/lib/etcd

1.1.12 On the etcd server node, get the etcd data directory, passed as an argument --data-dir,
from the command 'ps -ef | grep etcd'.
Run the below command (based on the etcd data directory found above).
For example, chown etcd:etcd /var/lib/etcd

1.1.19 Run the below command (based on the file location on your system) on the control plane node.
For example,
chown -R root:root /etc/kubernetes/pki/

References:

Feel free to add your thoughts and experiences, Happy Learning! 😄

k9s - manage your Kubernetes cluster and it’s objects like a pro!

Sunny Bhambhani — Sat, 30 Nov 2024 06:56:46 +0000

Introduction
k9s is a terminal based GUI to manage any Kubernetes(k8s) cluster. Using this single utility, we can manage, traverse, watch all our Kubernetes objects.

More information around k9s can be found here: https://k9scli.io/

We will dive a bit into k9s and see how it can help us in our day-to-day life, how we can get started, etc.

Features
Before we go into the example on how it can help, what it can do for us, let's see some of its features, it has a ton of features, but we will focus on the ones which can help us in our day-to-day activities:

Shows a helicopter view of all the Kubernetes objects.
Easily we can move from one namespace to another.
Watch/observe the state of Kubernetes objects using pulses.
Get the tree kind of structure using xrays to identify objects co-relation.
Manage the objects directly from the CLI, operations like deleting, editing, restarting is just one button away.
Interestingly using a single button, you can get shell access to the pods.
Check the logs from any of the pod without actually remembering the name of the pod, just select the pod and the logs are just one button away.
Sanitize/clean up all the pods which are in completed/error state.
Works with any flavor of Kubernetes minikube, Elastic Kubernetes Service(EKS), Azure Kubernetes Service(AKS), Google Kubernetes Engine(GKE), etc.
.... and a lot more.

Installation

Its installation is straight forward and is available for almost all the platforms.
Refer https://k9scli.io/topics/install/ for more details.
We will be doing this on Ubuntu.

Download the latest version of k9s binary, you can use the same URL/version if required or get the latest version released and update below URL accordingly.

For latest version, refer: https://github.com/derailed/k9s/releases

$ wget https://github.com/derailed/k9s/releases/download/v0.32.7/k9s_linux_amd64.deb

Install it using apt package manager.

$ sudo apt install ./k9s_linux_amd64.deb

Once done it will get the k9s binary installed here: /usr/bin/k9s.

HOWTO

Launch k9s

$ k9s

Once k9s is launched you will be presented with the beautiful layout with lots of options.

Press 0 to view all the pods from all the namespaces.

Navigate between Kubernetes objects
Navigating is pretty easy; it is more or less like vi or vim.

For example, if you want to view deployments then press : it will bring the cursor to a text area where you can write the object which you are interested in, for instance in current example deployments.
The short forms too are accepted i.e. deploy in case of deployment, svc in case of service, etc.
This also supports crds :)

Pods management

Press : and type in pods, it will show you all the pods in selected namespace, if you want to see the pods from all namespaces press 0.

There are times where we have to delete the pods on the fly. Just select the pod and press ctrl+d and you are done.

There are times when you have to login to a pod. Just select the pod and press s and you are done. If you want to exit, press ctrl+d or type in exit.

Want to describe the pod details, press d.

Want to get the yaml output of the Kubernetes objects, just select the object and press y.

Want to see the logs of a specific pod, select the pod and press l.

You can even do port-forward from the same UI, isn't it cool :), Just press shift+f and if you want to see any existing port-forward are present or not, press f.

Sanitize/Clean up pods: For instance in below screen dump we can see there are 2 pods which are in Completed and we want to clean them up.

Just press z, you will be asked "if you are sure", type in "Yes Please!" and the job is done.

XRAY
Xray gives you a great detail in terms of co-relation between k8s objects, it basically provides a tree like structure.

Pulses
Pulses give a great dashboard to see what exactly is happening in your cluster, what all objects are there, health of your objects, etc?

NOTE: Make sure metrics-server is installed and running in the cluster otherwise you won't see proper results. In my case it was not installed, therefore it was just stating blank results.

After installing metrics-server and launching pulses it shows an awesome dashboard.

The great part about it is, it shows the current status of your cluster.
It even shows what all objects are healthy and what all objects are unhealthy.
Legend: GREEN/HEALTHY & YELLOW/UNHEALTHY

References:

Feel free add your thoughts, Happy learning :)

Helmister: Updated the logic to allow custom path

Sunny Bhambhani — Thu, 28 Nov 2024 07:53:31 +0000

Its a follow-up post on helmister, refer here for more details: https://dev.to/aws-builders/installing-multiple-helm-charts-in-one-go-approach-3-using-simple-bash-utility-2795

Updated the logic allowing users to specify a custom path for config.yaml.
Log entry: [2024-11-27 12:30:06] [INFO] Generic/common values based on /home/sunny/config.yaml file.
-f | --file
Helps you to mention explicit/specific config.yaml. If not provided it will consider the default one which is placed in the base folder.
Example: ./helmister install -f /opt/app/config-dev.yaml

Installing multiple helm charts in one go [Approach 3 - using simple bash utility]

Sunny Bhambhani for AWS Community Builders ・ Jul 23 '24

#helm #kubernetes #devops #eks

sftp refresher - 101

Sunny Bhambhani — Tue, 29 Oct 2024 11:34:22 +0000

This is just a very basic refresher around sftp server, I know there are lots of other and on-cloud solutions available for file transfer and sftp server is rarely used.

But since I got a query yesterday about it, I thought to create a small write-up :)

sftp server

There was a time when sftp was widely used for secure file transfers, though now there are many options available, but this used to be a hero service then and was commonly asked during interviews.
The name itself expands to 'Secure File Transfer Protocol'.
This allows for a secure file transfer over SSH (On port 22).
It is quite fast and efficient.

Pre-requisites

Any linux machine with internet access (here I am using Ubuntu).

HOWTO

Update the package lists on your Ubuntu(debian based) machine.

$ apt update

Install ssh/openssh-sftp-server/vim(if not present).

$ apt install ssh openssh-sftp-server vim -y

Edit the sshd config file and append below details

$ vim /etc/ssh/sshd_config

Match group sftp
ChrootDirectory /home
X11Forwarding no
AllowTcpForwarding no
ForceCommand internal-sftp

Match group sftp: This statement states below settings will be applicable to all the users who belong to sftp group.
ChrootDirectory: This basically changes the root directory to mentioned path.
X11Forwarding: This disables X11 forwarding (this is enabled in some GUI use-cases) and for sftp this is not required.
AllowTcpForwarding: Disables TCP forwarding.
ForceCommand: This forcefully mandates that the used of this group should be only allowed to use sftp and nothing else.

Restart the service, you can use this way or use systemctl based on your distribution.

$ /etc/init.d/ssh restart

Add a new group called sftp to the system.

$ addgroup sftp

Create a new user with sftp group attached to it.

$ useradd -m sunny -g sftp

Change the password of the newly created user.

$ passwd sunny

Change the permission of the directory to USER/OWNER only.

$ chmod 700 /home/sunny/ -R

Try connecting to sftp server

$ sftp sunny@HOST_IP

Enter the password and try using get and put commands.

*Feel free to add any details which I might have missed, happy learning :) *

Quick guide to install minikube, kubectl, helm and enable ingress controller on Ubuntu

Sunny Bhambhani — Mon, 21 Oct 2024 12:39:29 +0000

This is just a quick guide to install minikube and enable ingress controller on Ubuntu.

The purpose of this post is to consolidate all the commands in one place for quick reference.

Minikube is a lightweight Kubernetes cluster that enables you to run Kubernetes clusters locally. It's ideal for learning Kubernetes while developing and testing on a single machine.

More information about minikube can be found here: https://minikube.sigs.k8s.io/docs/

Install minikube (Ref: https://minikube.sigs.k8s.io/docs/start)

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube && rm minikube-linux-amd64

Install docker (Ref: https://docs.docker.com/engine/install/ubuntu/)

sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

Add current user to docker group

sudo usermod -aG docker $USER && newgrp docker

Install kubectl (Ref: https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
sudo chmod +x kubectl
sudo mv kubectl /usr/local/bin
alias k=kubectl

Install helm (Ref: https://helm.sh/docs/intro/install/)

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Start minikube

minikube start
# Once started
k get nodes
# To get a list of underlying nodes
k get pods -A
# To get a list of all the pods from all namespaces

Get minikube's IP and enable ingress controller

minikube ip
minikube addons enable ingress

Installing multiple helm charts in one go [Approach 3 - using simple bash utility]

Sunny Bhambhani — Tue, 23 Jul 2024 10:34:55 +0000

In this article, we will be talking about Approach 3 i.e. how to get multiple helm charts installed using a simple bash utility.

If you haven't read the previous article where I discussed other approaches, feel free to read it over.

Motive

Why I thought of a simple bash utility?

In some of the air-gapped environments it is sometimes a bit difficult to use the tools/utilities available because moving things inside an air-gapped environment is a challenge.
Some of the environments are so secure that one may need to follow a whole process of getting all the security clearances and approval before using a tool/utility, which altogether is a nightmare.
I chose bash, the reason being it is pretty common among engineers and it is easily understandable.
The source code can be found here: https://github.com/sunnybhambhani/helmister
You can copy it, and tweak it based on your requirements.

PS: I chose a name for this utility as well helmister or helm-minister but you can call it whatever you want 🙂

Prerequisites

A running Kubernetes cluster with proper permissions, it can be any flavor of Kubernetes minikube, Elastic Kubernetes Service(EKS), Azure Kubernetes Service(AKS), Google Kubernetes Engine(GKE), etc.
kubectl and helm installed on your machine to interact with the Kubernetes cluster.
- Ref: https://kubernetes.io/docs/tasks/tools/
- Ref: https://helm.sh/docs/intro/install/
A clone of helmister repository.
yq(required) and cowsay(optional) packages installed on your machine.

Helmister

This is a small bash utility that can help to install and uninstall multiple helm charts in one go. The idea is inspired by helmfile.
Its usage is quite simple, just run the utility followed by the option like install or uninstall i.e. ./helmister [install/uninstall].
Under the hood, it calls helm binary. Therefore it is kind of a wrapper around helm.
It consumes a config.yaml file which contains all the necessary details about the releases, and common parameters.
This supports both oci:// and https:// helm registries.
PS: In the future, I am planning to add some more options/functionalities to this.

Directory structure

.
├── config.yaml
├── helmister
├── logs
│   ├── archive
│   │   └── helmister_20240227_170201.tar.gz
│   └── helmister.log
├── README.md
└── values
    ├── argo-cd.yaml
    └── nginx-values.yaml

helmister, this is the script written in bash you can just cat and see what all things it contains.
config.yaml, this is the main configuration file or you can call the state file which contains the list of all releases you want to install in a cluster, plus it also contains some additional key:value pairs that are generic and common across all the releases. I first kept this configuration file in csv format, but later decided to convert it to yaml because it is more readable.
logs, this is a directory that holds the logs of this utility, it contains the information about the execution of the last iteration, plus any archived/past logs (if required for reference).
README.md, contains bit of a documentation about this utility, and what options are present.
values, this is the directory where all the values files are placed (it can be anywhere in your system but for simplicity, I have placed them in the same directory).
Now let's talk about config.yaml which is the main ingredient.

config.yaml

dry_run: false
create_namespace: true
wait: false
timeout: false # If true, defaults to 20 mins
charts:
  - release_name: nginx
    chart_name: nginx
    chart_repo: oci://registry-1.docker.io/bitnamicharts
    values_file: values/nginx-values.yaml
  - release_name: argocd
    chart_name: argo-cd
    chart_repo: https://argoproj.github.io/argo-helm
    values_file: values/argo-cd.yaml
    version: 6.4.0
    namespace: argo-cd

The initial key:value pairs are common across all the releases.
charts, is an array that contains a list of what all releases need to be installed in a Kubernetes cluster.
dry_run, it is a boolean [true or false], and if true, none of the Kubernetes resources will be created it will just do a dry_run.
create_namespace, it is a boolean [true or false], and if true it will automatically create a namespace for the release specified in the charts array.
wait, it is a boolean [true or false], and if true it acts similar to helm --wait wherein the shell will be kept occupied until all the Kubernetes resources are created.
timeout, it is a boolean [true or false], and if true it acts similar to helm --timeout=20m i.e. if all the resources are not created within 20 mins the execution will fail. By default, I have kept the timeout as 20 minutes which is more than enough.
charts, it is an array that contains details around individual releases. Except for version and namespace all the key:value pairs are mandatory.
- release_name, this is the name of the release.
- chart_name, this is the name of the chart that needs to be installed.
- chart_repo, this is the helm registry where the chart is located. It can be any oci:// or https:// registry.
- values_file, which contains the path of the values file for individual releases.
- version, this is the version of the chart that needs to be installed. This is optional and if not provided, It will consider the latest chart version.
- namespace, this is where the chart will be installed. This is optional as well and if not provided it will be installed in the default namespace.

Let's see this in action

I will use the same config.yaml which will install one helm release from an oci:// registry in default namespace (since I haven't specified any namespace for that release) and another one from https:// registry in argo-cd namespace. Note the version as well for nginx, there as well I haven't specified any version, it will pick the latest available one automatically.
This is my cluster's current status.

$ k get ns
NAME              STATUS   AGE
default           Active   28s
kube-node-lease   Active   28s
kube-public       Active   29s
kube-system       Active   29s

$ helm list -A
NAME    NAMESPACE       REVISION        UPDATED STATUS  CHART   APP VERSION

$ ./helmister install
[2024-07-23 15:02:47] [INFO] Using file: config.yaml
 ____________________________
< Helmister, Install charts! >
 ----------------------------
   \
    \
        .--.
       |o_o |
       |:_/ |
      //   \ \
     (|     | )
    /'\_   _/`\
    \___)=(___/

[2024-07-23 15:02:47] [INFO] Generic/common values based on config.yaml file
[2024-07-23 15:02:48] [INFO] Dry Run: false
[2024-07-23 15:02:48] [INFO] Create Namespace: true
[2024-07-23 15:02:48] [INFO] Wait: false
[2024-07-23 15:02:48] [INFO] Timeout: false
[2024-07-23 15:02:48] [INFO] ****************************
[2024-07-23 15:02:48] [INFO] Chart specific values based on config.yaml file
[2024-07-23 15:02:48] [INFO] Release Name: nginx
[2024-07-23 15:02:48] [INFO] Chart Name: nginx
[2024-07-23 15:02:48] [INFO] Chart Repo: oci://registry-1.docker.io/bitnamicharts
[2024-07-23 15:02:48] [INFO] Values File: values/nginx-values.yaml
[2024-07-23 15:02:48] [INFO] Version: null
[2024-07-23 15:02:48] [INFO] Namespace: null
[2024-07-23 15:02:48] [INFO] ****************************
[2024-07-23 15:02:48] [INFO] Installing nginx with release name nginx from oci://registry-1.docker.io/bitnamicharts with version null using values file: values/nginx-values.yaml in default namespace
...
...
OUTPUT TRIMMED

Actual stdout output can be found here.
Here you will see that it provides all the minute details of what exactly it is doing.
For example: Generic values
- Here I have marked create_namespace as true because I don't already have argo-cd namespace wherein I want to install argocd release.

[2024-07-23 15:02:47] [INFO] Generic/common values based on config.yaml file
[2024-07-23 15:02:48] [INFO] Dry Run: false
[2024-07-23 15:02:48] [INFO] Create Namespace: true
[2024-07-23 15:02:48] [INFO] Wait: false
[2024-07-23 15:02:48] [INFO] Timeout: false

Next, you will see, chart/release specific values of all the items in charts array one by one:

[2024-07-23 15:03:31] [INFO] Chart specific values based on config.yaml file
[2024-07-23 15:03:31] [INFO] Release Name: argocd
[2024-07-23 15:03:31] [INFO] Chart Name: argo-cd
[2024-07-23 15:03:31] [INFO] Chart Repo: https://argoproj.github.io/argo-helm
[2024-07-23 15:03:31] [INFO] Values File: values/argo-cd.yaml
[2024-07-23 15:03:31] [INFO] Version: 6.4.0
[2024-07-23 15:03:31] [INFO] Namespace: argo-cd

Here I have explicitly added an additional check that will check if the pods are up and healthy (this is just specific to pods it won't consider any other k8s objects).
It will continuously check for next 20mins and will check every 5 secs. If the pods are still in non-running state it will terminate the process.

[2024-07-23 15:03:47] [WARN] Pods for release argocd are not yet in Running state, checking again in 5 seconds

Once all the pods are in running state and it has deployed all the releases it will show a successful message.

[2024-07-23 15:05:20] [INFO] All actions completed successfully

Once this is done, if required you can see the logs as well from the logs directory. A sample can be found here.
And here is my cluster's current status now:

$ helm list -A
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
argocd  argo-cd         1               2024-07-23 15:03:34.46715742 +0530 IST  deployed        argo-cd-6.4.0   v2.10.1
nginx   default         1               2024-07-23 15:02:53.590367521 +0530 IST deployed        nginx-18.1.5    1.27.0

$ k get ns
NAME              STATUS   AGE
argo-cd           Active   58m
default           Active   62m
kube-node-lease   Active   62m
kube-public       Active   62m
kube-system       Active   62m

$ k get pods -n argo-cd
NAME                                                READY   STATUS    RESTARTS   AGE
argocd-application-controller-0                     1/1     Running   0          58m
argocd-applicationset-controller-68cd75bc89-d4q8x   1/1     Running   0          58m
argocd-dex-server-84b5bbdcbf-c4qv2                  1/1     Running   0          58m
argocd-notifications-controller-7bc55c495d-xz6gt    1/1     Running   0          58m
argocd-redis-5d5cdcfd54-2rx26                       1/1     Running   0          58m
argocd-repo-server-756ff4cd7d-rpvh7                 1/1     Running   0          58m
argocd-server-7587d49b7b-9h47l                      1/1     Running   0          58m

$ k get pods -n default
NAME                     READY   STATUS    RESTARTS   AGE
nginx-569b6bc698-48qqt   1/1     Running   0          59m
nginx-569b6bc698-9qzpv   1/1     Running   0          59m
nginx-569b6bc698-wflrj   1/1     Running   0          59m

If you want to uninstall it, simply do:

$ ./helmister uninstall

It will get everything cleaned for all the releases that are specified in config.yaml.
The sample log can be found here.

Feel free to use it, and tweak it based on your requirements.

I will soon add some more functionalities to it.

Happy learning!

References:

Google/Stackoverflow/Linux man pages/etc

Update multiple Kubernetes objects/configmaps in one go!

Sunny Bhambhani — Wed, 22 May 2024 12:54:25 +0000

There may be cases wherein we just need to update a Kubernetes configmap or any other Kubernetes object based on our requirements.

And let's say it's just one or two configmaps(here we will be talking about configmaps just for simplicity but its applicable for other Kubernetes objects as well) then that's fine, right? We can simply do the following:

kubectl edit -n NAMESPACE_NAME configmap CONFIGMAP_NAME
Update it based on our requirements.
Save it.

But think about a scenario, wherein we need to update +100 configmaps with the same values how to do it?

We cannot update or edit each and every object since that would be a bit tedious and will be error-prone as well. Plus there might be chances we might skip some of them.

So, the best way to do it is by using patch command in an automated way.

Refer: https://kubernetes.io/docs/reference/kubectl/generated/kubectl_patch/ for more details about it.

Let's see this in action:

I have a very minimal configmap which contains some DATABASE details.

apiVersion: v1
data:
  DATABASE_HOST: "api01.ybdb.io"
  DATABASE_NAME: "api01"
  DATABASE_PORT: "5433"
kind: ConfigMap
metadata:
  name: api01-configmap
  namespace: default

Now, let's assume there is an update instead of port number 5433. Our database now listens on port 5444.
We can easily accomplish this using kubectl edit but let's see patch in action:

$ kubectl patch configmap api01-configmap -n default -p '{"data":{"DATABASE_PORT":"5444"}}'
configmap/api01-configmap patched
$ kubectl get cm -n default -o yaml api01-configmap
apiVersion: v1
data:
  DATABASE_HOST: api01.ybdb.io
  DATABASE_NAME: api01
  DATABASE_PORT: "5444"
kind: ConfigMap
metadata:
  creationTimestamp: "2024-05-22T10:06:39Z"
  name: api01-configmap
  namespace: default
  resourceVersion: "67651"
  uid: 2d49a0ea-6910-420c-b2bf-00e42e6c08d7

The command is pretty straight forward kubectl patch configmap api01-configmap -n default -p '{"data":{"DATABASE_PORT":"5444"}}'
-n is to specify the namespace name where the configmap resides.
-p is to specify what patch/update we want in the Kubernetes object.
Here, under data: we are updating a key called DATABASE_PORT: to 5444.
Note: If in case DATABASE_PORT: key doesn't exist, it will create that key for us.

Assume you want to see the output as well in the same command then add -o or --output flag with the type [json,name,yaml,etc].

$ kubectl patch configmap api01-configmap -n default -p '{"data":{"DATABASE_PORT":"5444"}}' -o yaml
apiVersion: v1
data:
  DATABASE_HOST: api01.ybdb.io
  DATABASE_NAME: api01
  DATABASE_PORT: "5444"
kind: ConfigMap
metadata:
  creationTimestamp: "2024-05-22T10:47:41Z"
  name: api01-configmap
  namespace: default
  resourceVersion: "69416"
  uid: a303bd0e-7f61-49c0-b671-06649ecdf999

But there is another catch, you can also not run the patch command +100 times, I mean definitely you can but again that might be error-prone too.

So, in that case we can write a simple reusable bash script that can do the job for us and can help us to update +100 configmaps in one go.

It can be in any language but I have chosen bash, since it's pretty common.

Quick rundown of the script:

Here I have three variables, NAMESPACE (where the object resides), CONFIGMAP_FILE (input file which contains a list of all the configmaps that need to be updated), DATA (a set of key:value pair which needs to be replaced/added).
Here I am creating log files as well (update_cm.log - contains generic readable information, patch.log - contains output of patch command).
As of now, I just have added action as add but we can have other actions as well like replace or remove (based on our requirements).

#!/bin/bash

NAMESPACE="default"
CONFIGMAP_FILE="configmaps"
DATA='{"data":{"DATABASE_PORT":"5444"}}'

rm patch.log update_cm.log

if [ ! -f "$CONFIGMAP_FILE" ]; then
  echo "File $CONFIGMAP_FILE not found."
  exit 1
fi

echo `date` >> update_cm.log
echo "----------------------------" >> update_cm.log

action=$1

if [ "$action" == "add" ]; then
  echo "Action: $action"
  echo "Action: $action" >> update_cm.log
  while IFS= read -r cm; do
    echo "Updating ConfigMap: $cm"
    echo "Updating ConfigMap: $cm" >> update_cm.log
    kubectl patch configmap "$cm" -n "$NAMESPACE" -p $DATA >> patch.log 2>&1
  done < "$CONFIGMAP_FILE"

else
  echo "Invalid action, please specify add".
  exit 1
fi

echo "----------------------------" >> update_cm.log

Feel free to use it, tweak it, based on your requirements. This script is for configmaps but similar logic we can write for other objects as well if required.

Happy learning :)

References:

Google.
Kubernetes docs (https://kubernetes.io/docs/home/).
kubectl explain

Disclaimer:

We should not manually update/patch the Kubernetes objects, this should be done when absolutely necessary or needed on the fly.
Ideally these all values should come from the version control system wherein the values files are maintained.

PS: This works with any flavor of Kubernetes minikube, Elastic Kubernetes Service(EKS), Azure Kubernetes Service(AKS), Google Kubernetes Engine(GKE), etc.