DEV Community: Ashiqur Rahman

Enforcing Kubernetes Probes with a Custom Admission Webhook

Ashiqur Rahman — Mon, 16 Jun 2025 13:17:31 +0000

Introduction

Container health checks via liveness and readiness probes are critical for resilient workloads in Kubernetes. However, it’s easy for teams to forget adding them, leading to issues that are only discovered in production.

In this post, I’ll walk through building a custom Kubernetes Admission Controller that enforces the presence of liveness and readiness probes on all pods. We’ll cover everything: why, how, code, testing, and deployment.

Why Enforce Probes?

Resilience: Probes help Kubernetes automatically restart failing pods and avoid routing traffic to unhealthy ones.

Standardization: Ensures organizational best practices are followed without relying on manual checks or code reviews.

Automation: Moves policy enforcement to the cluster level, catching issues before they reach production.

Design Overview

Our Validating Admission Controller runs as a K8s deployment. It intercepts all pod creation requests in it's cluster and:

Checks for startup, readinessProbe and livenessProbe on every container.
Rejects pods missing atleast one required probe, with a clear error message. You can configure which probes you want to enforce by setting the environment variable ENFORCE_PROBES (default: liveness,readiness). You can also exclude some namespaces using the environment variable EXCLUDE_NAMESPACES (default: kube-system,kube-public)

The flow:

User creates/updates a pod.
K8s API calls our webhook.
Webhook inspects the pod spec.

If probes are missing, request is denied. The pod creation fails with an error from our custom admission controller.

Demo

Pod in `default` namespace with no probes

First, let's try to create a pod with no probes defined in the default namespace:

Looks like our admission webhook intercepted it and blocked the pod creation. Let's confirm that from the logs:

Pod in `kube-system` namespace with no probes defined

Now, we try to create a pod with no probes defined in the kube-system namespace:

Looks like the pod creation succeeded this time, which is the expected behaviour. This is because, we have excluded the kube-system namespace from the probe enforcer. Let's verify from the logs:

Pod in `default` namespace with both liveness and readiness probes defined

The pod creation succeeded. Let's verify from the logs again:

Installation

Pre-requisites

A Kubernetes cluster with cert-manager installed. https://cert-manager.io/docs/installation/#default-static-install

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.18.0/cert-manager.yaml

Now, apply the manifests required for installing the admission controller:

kubectl kustomize https://github.com/ashiqursuperfly/k8s-probe-enforcer-ac/k8s | kubectl apply -f -

Once all the manifests are applied, our custom admission webhook is ready to intercept all pod create/update requests. Try creating a pod with a liveness/readiness probe missing and see the magic!

Implementation

The complete source code is hosted on Github: https://github.com/ashiqursuperfly/k8s-probe-enforcer-ac
Here's the core logic that validates an incoming pod create/update request:

if cfg.ExcludeNamespaces[pod.Namespace] {
    log.Printf("Namespace '%s' is excluded from validation. Allowing pod '%s'.", pod.Namespace, pod.Name)
    respondAdmission(w, review, true, "")
    return
}

containersMissingProbes := map[string][]string{}
for _, c := range pod.Spec.Containers {
    for _, probe := range cfg.EnforceProbes {
        switch probe {
        case "readiness":
            if c.ReadinessProbe == nil {
                containersMissingProbes[c.Name] = append(containersMissingProbes[c.Name], "readinessProbe")
            }
        case "liveness":
            if c.LivenessProbe == nil {
                containersMissingProbes[c.Name] = append(containersMissingProbes[c.Name], "livenessProbe")
            }
        case "startup":
            if c.StartupProbe == nil {
                containersMissingProbes[c.Name] = append(containersMissingProbes[c.Name], "startupProbe")
            }
        }
    }
}

if len(containersMissingProbes) == 0 {
    log.Printf("Pod '%s' passed validation. All required probes are present.", pod.Name)
    respondAdmission(w, review, true, "")
    return
}

// Prepare detailed error message
var failMsg string
for container, probes := range containersMissingProbes {
    failMsg += fmt.Sprintf("Container '%s' missing: %v; ", container, probes)
}
log.Printf("Pod '%s' rejected. %s", pod.Name, failMsg)

respondAdmission(w, review, false, failMsg)

Docs

Here are some of the official kubernetes documentation that helped me come up with my solution:

https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/
https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/
https://github.com/ashiqursuperfly/k8s-probe-enforcer-ac (Please leave a star on the repo if you found this useful!)

Handling mapping changes in Elasticsearch and reindexing

Ashiqur Rahman — Thu, 20 Mar 2025 00:34:16 +0000

Elasticsearch is a powerful search engine that allows for flexible and efficient data indexing and searching. However, managing changes in the data schema, known as mappings, can be challenging. This guide will walk you through handling mapping changes and the reindexing process in Elasticsearch.

Understanding Mappings

Mappings in Elasticsearch define how documents and their fields are stored and indexed. They determine the data types (e.g., text, keyword, integer) and how fields should be analyzed. Properly managing mappings is crucial for ensuring efficient search operations and data integrity.

Why Mapping Changes Occur

Mapping changes might be necessary due to:

Schema Evolution: As your application evolves, new fields might need to be added, or existing fields might require changes.
Optimization: Improving search performance by changing field types or adding analyzers.
Error Correction: Fixing mistakes in the initial mapping setup.

Challenges with Mapping Changes

Elasticsearch does not allow direct modification of existing field mappings. This limitation ensures data consistency and prevents potential conflicts. Therefore, any changes to mappings require careful planning and execution.

Steps to Handle Mapping Changes

Plan the Changes: Identify the fields that need changes and determine the new mapping requirements. Consider the impact on existing data and search queries.
Create a New Index: Since mappings cannot be changed directly, you need to create a new index with the desired mappings. This new index will eventually replace the old one.
Reindex the Data: Reindexing involves copying data from the old index to the new index. Elasticsearch provides a _reindex API to facilitate this process.

   POST _reindex
   {
     "source": {
       "index": "old_index"
     },
     "dest": {
       "index": "new_index"
     }
   }

Update Aliases: Once reindexing is complete, update any aliases pointing to the old index to point to the new index. This step ensures that applications querying the index continue to function without changes.

   POST /_aliases
   {
     "actions": [
       { "remove": { "index": "old_index", "alias": "my_index" }},
       { "add": { "index": "new_index", "alias": "my_index" }}
     ]
   }

Verify Data Integrity: After reindexing, verify that the data in the new index is correct and that all mappings are as expected. Perform searches to ensure that the new mappings work as intended.
Delete the Old Index: Once you are confident that the new index is functioning correctly, you can safely delete the old index to free up resources.

   DELETE /old_index

Best Practices

Backup Data: Always backup your data before making significant changes.
Test Changes: Use a staging environment to test mapping changes and reindexing processes.
Monitor Performance: After reindexing, monitor the performance of your Elasticsearch cluster to ensure that the changes have not introduced any issues.

By following these steps and best practices, you can effectively manage mapping changes in Elasticsearch and ensure a smooth reindexing process.
The cover image in this post was taken from this OG reddit thread

A guide to debugging a Kubernetes deployment

Ashiqur Rahman — Sun, 16 Mar 2025 19:29:30 +0000

A Guide to Debugging a Kubernetes Deployment

Debugging a Kubernetes deployment can be a challenging task, especially for those new to container orchestration. After working extensively with all three major managed Kubernetes offerings (EKS, GKE and AKS) for the last three years, I aim to provide a structured approach to identifying and resolving issues within your Kubernetes deployments in this post. These are just the things I check out quickly, whenever an alert pops up on PagerDuty. These proved to be sufficient in my experience to get to the root cause of a failing deployment quickly. As the technologies built in and around Kubernetes evolve, I plan to update this post as needed.

Understanding the Basics
Common Issues and Solutions
Tools and Commands
Best Practices
Conclusion

Understanding the Basics

Before diving into debugging, it's essential to understand the basic components of a Kubernetes deployment:

Pods: The smallest deployable units in Kubernetes.
Deployments: Controllers that manage the desired state of pods.
Services: Abstract ways to expose an application running on a set of pods.
ConfigMaps and Secrets: Methods for managing configuration data and sensitive information.

Understanding these components will help you pinpoint where issues may arise.

Common Issues and Solutions

1. Pods Not Starting

Check Pod Status: Use kubectl get pods to check the status of your pods. Look for pods in Pending or CrashLoopBackOff states.
Inspect Events: Use kubectl describe pod <pod-name> to view events and error messages that can provide clues.

2. Image Pull Failures

Verify Image Name: Ensure the image name in your deployment is correct.
Check Image Registry: Make sure the image is available in the specified registry and that your Kubernetes nodes have access to it.

3. Configuration Errors

Validate ConfigMaps and Secrets: Use kubectl describe configmap <name> and kubectl describe secret <name> to ensure they are correctly configured.
Check Environment Variables: Ensure that the environment variables in your deployment manifest are correctly set.

4. Networking Issues

Service Discovery: Verify that services are correctly defined and that DNS resolution is working.
Network Policies: Check if network policies are blocking traffic between pods.

Tools and Commands

kubectl logs: Retrieve logs from a pod to diagnose issues.

  kubectl logs <pod-name>

kubectl exec: Execute commands inside a running pod for troubleshooting.

  kubectl exec -it <pod-name> -- /bin/sh

kubectl describe: Provides detailed information about a resource.

  kubectl describe pod <pod-name>

kubectl get events: Check for events that might indicate issues.

  kubectl get events

Best Practices

Resource Requests and Limits: Define resource requests and limits to ensure pods have the necessary resources and to prevent resource exhaustion.
Health Checks: Implement liveness and readiness probes to automatically handle unhealthy pods.
Logging and Monitoring: Use centralized logging and monitoring tools like Prometheus and Grafana for better visibility.

Conclusion

Debugging Kubernetes deployments requires a systematic approach and familiarity with Kubernetes components and tools. By following this guide, you can effectively troubleshoot and resolve common issues, ensuring your applications run smoothly in a Kubernetes environment.

Remember, practice makes perfect. The more you work with Kubernetes, the more adept you'll become at identifying and solving deployment issues.

How to deploy and manage a RabbitMQ cluster on Amazon EKS using Terraform and Helm

Ashiqur Rahman — Tue, 16 Jan 2024 15:14:51 +0000

In this blog post, we will dive into deploying and managing RabbitMQ on Kubernetes with the powerful combination of Terraform and Helm. As businesses increasingly embrace microservices architecture, efficient message queuing becomes paramount, and RabbitMQ emerges as a robust solution. In this guide, we'll walk you through the step-by-step process of setting up RabbitMQ on a Kubernetes cluster, leveraging the infrastructure-as-code capabilities of Terraform and the Kubernetes package manager Helm.

When should you consider deploying RabbitMQ on Kubernetes

There are a few different ways you can leverage RabbitMQ for

You can deploy RabbitMQ on a single cloud VM (e.g: on an EC2 machine)

This option can be suitable for you in some cases but it has the obvious disadvantage of having to patch, secure, and monitor RabbitMQ all by yourself.
Also, a single-node RabbitMQ may not be the most optimal solution for you since, it will not be highly available, fault-tolerant, or scalable.
Furthermore, you would probably need a few RabbitMQ instances for different services. Let's say you need ten independent RabbitMQ instances for ten independent services in your microservices. What if this number (10) can change quite rapidly? In these sorts of circumstances, it would be increasingly difficult to operate these instances for your DevOps team.

Use a cloud provider-managed RabbitMQ e.g. AmazonMQ.

While AmazonMQ does most of the heavy lifting for you out of the box, you might want to avoid it due to reasons such as

cost
avoiding a vendor lock-in

Deploy it on Kubernetes!

This is exactly what we are going to do today and we will see how effectively this solution deals with most of the problems we discussed in the other two options we discussed.

Challenges with deploying RabbitMQ on a Kubernetes cluster:

How should we persist RabbitMQ state?

We are gonna be using Kubernetes StatefulSets and PersistentVolumes to solve this problem. When a StatefulSet creates a pod, it automatically creates a PVC (persistent volume claim) for that pod based on the template defined in the StatefulSet's volumeClaimTemplates field. The PVC ensures that the pod has access to persistent storage, and the associated PV is dynamically provisioned or statically bound based on the storage class and other specifications. The StatefulSet ensures that each pod receives a unique identifier in the form of an ordinal index, and this identifier is used to create a unique PVC for each pod. Furthermore, StatefulSets provides stable network identities for pods. This is essential for stateful applications that might rely on predictable network addresses or hostnames. The stable network identity allows for services, applications, or other components to consistently locate and communicate with stateful pods, even if they are rescheduled or re-created.

How should the different nodes in a RabbitMQ cluster communicate with each other inside Kubernetes?

StatefulSets explained above already solves half of the problem and the other half is gonna be solved by Headless-Services. A headless service is a service without a cluster IP, and it is used to create DNS records for the pods it selects. Each selected pod gets its unique DNS record based on the service name. The RabbitMQ pods can directly communicate with each pod using its unique DNS name to register and de-register pods in the cluster.

Log lines when the process takes place:



2024-01-16 13:11:18.176142+00:00 [warning] <0.731.0> Peer discovery: node rabbit@dev-my-service-a-rabbitmq-1.dev-my-service-a-rabbitmq-headless.rabbitmq.svc.cluster.local is unreachable
2024-01-16 13:11:28.194766+00:00 [info] <0.544.0> node 'rabbit@dev-my-service-a-rabbitmq-1.dev-my-service-a-rabbitmq-headless.rabbitmq.svc.cluster.local' up
2024-01-16 13:11:33.074253+00:00 [info] <0.544.0> rabbit on node 'rabbit@dev-my-service-a-rabbitmq-1.dev-my-service-a-rabbitmq-headless.rabbitmq.svc.cluster.local' up

How should we automate the provisioning of as many RabbitMQ instances as our applications need?

We will write a Terraform module that will take a list of configurations for each required RabbitMQ instance. Luckily for us, we don't have to write the Kubernetes yaml configurations since the helm charts by Bitnami does a great job of doing all the things we discussed above. All we need to do is leverage Terraform Helm Provider and deploy the chart with the required values for our use case.

Terraform module to provision a list of RabbitMQ instances

Defining the list of RabbitMQ instances each of our applications needs in a yaml file. Here's how it might look like:

instances.yaml



nlb_subnets: "subnet-0123456789a\\, subnet-0123456789b\\, subnet-0123456789c" 

instances:
  my-service-a-rabbit:
    username: service-a-admin
    password: aws-kms-encrypted-super-secret-password-a
    image_registry: docker.io
    image_repository: bitnami/rabbitmq
    image_version: 3.12.12
    replica_count: 3

  my-service-b-rabbit:
    username: service-b-admin
    password: aws-kms-encrypted-super-secret-password-b
    replica_count: 2

  my-service-c-rabbit:
    username: service-a-admin
    password: aws-kms-encrypted-super-secret-password-c
    replica_count: 2

  my-service-d-rabbit:
    username: service-b-admin
    password: aws-kms-encrypted-super-secret-password-d
    replica_count: 4

Here, nlb_subnets is a list of subnet IDs that we would require to create a Network Load Balancer. These can be public or private subnets depending on your load balancer's scheme. Here, we are using a Network Load Balancer as an access point for each RabbitMQ instance on the list. You might ask why we need a Network Load Balancer (not an Application Load Balancer)? Well, RabbitMQ clients communicate with RabbitMQ servers using the amqp:// protocol, and Application Load Balancers can only balance HTTP or HTTPS traffic. NLBs can load balance traffic at the TCP layer which does not depend on layer 7 protocols. Please note, we are using AWS Load Balancer Controller to provision the AWS load balancers.

Now, we need to make sure our Terraform module can read the yaml file name as a variable and parse it accordingly to set different values Helm would require.

variables.tf



variable "config_file" {
  description = "yaml file containing configuration for all rabbitmq instances"
}

main.tf



locals {
  config    = yamldecode(file("${path.root}/conf/${var.config_file}"))
  instances = local.config["instances"]
  subnets   = local.config["nlb_subnets"]
}

data "aws_kms_secrets" "decrypt_password" {
  for_each = local.instances
  secret {
    name    = "master_password"
    payload = each.value.password
  }
}

resource "helm_release" "rabbit" {
  for_each         = local.instances
  name             = each.key
  namespace        = lookup(each.value, "namespace", "rabbitmq")
  create_namespace = true
  timeout          = 600
  repository       = lookup(each.value, "chart_repository", "oci://registry-1.docker.io/bitnamicharts")
  chart            = "rabbitmq"
  version          = lookup(each.value, "chart_version", "12.6.2")

  set {
    name  = "auth.username"
    value = lookup(each.value, "username", "rabbit")
  }

  set {
    name  = "auth.password"
    value = lookup(data.aws_kms_secrets.decrypt_password, each.key, null) != null ? data.aws_kms_secrets.decrypt_password[each.key].plaintext["master_password"] : null
  }

  set {
    name  = "auth.erlangCookie"
    value = each.key
  }

  set {
    name  = "image.registry"
    value = lookup(each.value, "image_registry", "docker.io")
  }

  set {
    name  = "image.repository"
    value = lookup(each.value, "image_repository", "bitnami/rabbitmq")
  }

  set {
    name  = "image.tag"
    value = lookup(each.value, "image_version", "3.12.12")
  }

  set {
    name  = "replicaCount"
    value = lookup(each.value, "replica_count", 1)
  }

  set {
    name  = "service.type"
    value = "LoadBalancer"
  }

  set {
    name  = "service.annotations.service\\.beta\\.kubernetes\\.io/aws-load-balancer-type"
    value = "external"
  }

  set {
    name  = "service.annotations.service\\.beta\\.kubernetes\\.io/aws-load-balancer-nlb-target-type"
    value = "ip"
  }

  set {
    name  = "service.annotations.service\\.beta\\.kubernetes\\.io/aws-load-balancer-subnets"
    value = local.subnets
  }
}

Similarly, override any other values you might need from here: https://github.com/bitnami/charts/blob/main/bitnami/rabbitmq/values.yaml
Consider reading about the ErlangCookie which is very important for clustered RabbitMQ environments.

Now, we just need to invoke our module like so, ```hcl

module "rabbit" {
source = "../modules/rabbitmq-k8s" # make sure you keep the module files accordingly
config_file = "instances.yaml"
}


- Now, we just need to run Terraform plan and apply!
```bash


terraform plan -out=plan.out
terraform apply plan.out

Once the Terraform apply is complete our application clients can start communicating with the RabbitMQ instances using one of the following approaches:
I) The ClusterIP service DNS name (only works if your applications are also inside the same cluster)
II) The NLB hostname
III) If you don't prefer using the NLB hostnames, consider using External-DNS to attach a custom domain name

And that's all folks! Feel free to comment if you have any questions. If you found this post helpful, CLICK BELOW 👇

Optimize EC2 cost by scheduling instances using EventBridge scheduler and Lambda

Ashiqur Rahman — Mon, 03 Jul 2023 09:26:23 +0000

Scheduling EC2 instances is a crucial step in AWS cost optimization. Because EC2 instances are invoiced by the hour or second depending on their usage, running instances constantly even when they are not in use can quickly add up to excessive costs. Organizations can connect their use of EC2 instances with actual business needs by putting in place a well-thought-out scheduling strategy, ensuring that instances are only operational during necessary times. By minimizing idling time, eliminating pointless fees, and increasing the effective use of computing resources, this strategy enables significant cost reductions. Companies can successfully control their AWS spending while still meeting operational expectations by dynamically scheduling instances to turn on and off at precise times.

In one of my side hustles, I stumbled upon a situation where we need to provision some new EC2 instances for an accounting software. We already purchased some reserved instances for our web applications that need to be running 24/7 and did not want to buy any more of them. Since, the accounting software will only be used during 9-5 work hours, we decided to schedule the EC2 instances to run only during the active hours. In this post, I'll show you the easiest way (imo) to achieve this behaviour.

If you look it up on Google, you might find this cloudformation-template. However, when i really took a deeper dive into this, I felt like this is quite overkill for something so simple as stopping instances for specific hours in the day. So, i decided to write my own solution using just lambda and EventBridge-Scheduler.

Step 1: Lambda

import json
import boto3
from datetime import datetime

# operating hours are specified in Asia/BD GMT+6 time
config = {
    "frontend": {
        "instance_id": "i-0b395375beefed021b",
        "non_operating_hours": {
            "start": 2,
            "end": 6
        }
    },
    "backend": {
        "instance_id": "i-054281d348bbf103165",
        "non_operating_hours": {
            "start": 2,
            "end": 6
        }
    },
    "accounts": {
        "instance_id": "i-0fb10809264e10342",
        "non_operating_hours": {
            "start": 0,
            "end": 9
        }
    }
}

def get_instance_state(instance_id):
    ec2_client = boto3.client('ec2')
    response = ec2_client.describe_instances(InstanceIds=[instance_id])

    reservations = response['Reservations']
    if not reservations:
        print("No reservations found for the provided instance ID.")
        return

    instances = reservations[0]['Instances']

    if not instances:
        print("No instances found for the provided instance ID.")
        return

    return instances[0]['State']['Name']


def start_instance(instance_id):
    ec2_client = boto3.client('ec2')
    response = ec2_client.start_instances(InstanceIds=[instance_id])
    return response

def stop_instance(instance_id):
    ec2_client = boto3.client('ec2')
    response = ec2_client.stop_instances(InstanceIds=[instance_id])
    return response

def lambda_handler(event, context):
    current_hour = datetime.now().hour
    print(f'Executing on BD hour {current_hour}')

    results = []
    for instance_name, conf in config.items():
        instance_id = conf['instance_id']
        instance_state = get_instance_state(instance_id)

        start = conf['non_operating_hours']['start']
        end = conf['non_operating_hours']['end']

        if current_hour >= start and current_hour <= end:
            is_stopped = 'stop' in instance_state

            if not is_stopped:
                res = stop_instance(instance_id)

            results.append(
                {
                    'action': "Stop" if not is_stopped else f"Already {instance_state}",
                    'conf': conf
                }
            )

        else:
            is_running = 'run' in instance_state
            if not is_running:
                res = start_instance(instance_id)

            results.append(
                {
                    'action': "Start" if not is_running else f"Already {instance_state}",
                    'conf': conf
                }
            )        

    response = json.dumps(results)

    print(response)

    return {
        'statusCode': 200,
        'body': response
    }

Please note that, this lambda code may or may not be a basic chatgpt dump 🤐 and there is huge room for improvement :) Feel free to improve it yourself as needed.

Step 2: Create EventBridge schedule

AWS EventBridge supports cron expression based schedulers which happens to be just perfect for this use case. So, we decided to use it.
We can create the schedule either using Terraform, AWS CLI or AWS console.
For example, the terraform code will look sth like this,

module "eventbridge" {

  create_bus = false

  rules = {
    crons = {
      description         = "Trigger for EC2StartStop Lambda"
      schedule_expression = "rate(1 hours)"
    }
  }

  targets = {
    crons = [
      {
        name  = "EC2StartStopLambda-Target-Cron"
        arn   = your_lambda_function_arn
        input = jsonencode({ "job" : "cron-by-rate" })
      }
    ]
  }

Now, every 1 hour our scheduler will trigger the lambda function and our lambda function will determine whether an instance should be up during that hour or not. This solution takes about 5minutes to setup but has the potential to reduce your compute bills by a significant margin (in my case, we reduced about 600 USD per month).

If you found this post helpful, CLICK BELOW 👇

Automate Jenkins Job builds with Python

Ashiqur Rahman — Fri, 16 Jun 2023 09:36:12 +0000

We often come across a situation when we have to manually run a Jenkins job many times each possibly having different parameters. One way to do it is using the Jenkins UI and pressing the build with parameters button each time and passing on the parameter values each time. This can be quite tiresome and repetitive. Fortunately, I am too Lazy to do it like that xD. So, I found another way, we can trigger the Jenkins jobs using the Jenkins REST API. In this post, we will demonstrate how to design our Jenkins jobs effectively for this purpose and use python to make use of the Jenkins API to automate triggering a huge number of jobs and then take a chill pill :)

Step 1: Enable Triggering Jenkins jobs from remote scripts

Navigate to your Jenkins Job configuration page and scroll down to the Build Triggers option. Enable the Trigger builds remotely (e.g., from scripts) flag and put some secret string in the Authentication Token field.

Step 2: Parameterize Job -> GIT SCM (BRANCH, REPO ETC.)

We need to make our Job bash script read from parameters we passed in to the Job so that we can easily control these parameters from our python script. Furthermore, if there's a git repo/branch our Job needs to checkout we can easily control that from python as well.
Enable the This project is parameterized flag from the Jenkins job configuration page and add the required parameters like below,

Jenkins supports various different types of parameters including Strings, Boolean, Choices, File etc. Make sure you are choosing them appropriately for your choice. In my case for example, I had to deploy a bunch of Ansible playbooks to a Kubernetes cluster so, I needed the following parameters:



PLAYBOOK - name of the yaml ansible playbook file
GIT_REPO_ID - git repo to checkout
GIT_BRANCH_NAME - branch name of the repo (particularly useful when you are testing/reviewing something on a feature branch)
HELM_REPOSITORY - helm repository name that my ansible playbooks needs

Now, access these parameters wherever you need inside the Jenkins config using the ${PARAM_NAME} syntax.

For example my job bash script looks like the one just below where, I am utilizing the PLAYBOOK and HELM_REPOSITORY parameters.



#!/bin/bash
set -xe

pip3 install --quiet -r requirements.txt

cd ansible

ANSIBLE_FORCE_COLOR=true ansible-playbook -e "helm_repository=${HELM_REPOSITORY}" \
playbooks/${PLAYBOOK} -vvvv

Here's another example where I am utilizing the GIT SCM parameters,

Step 3: Generate Jenkins Crumb

Jenkins API will need us to pass in a Jenkins-Crumb header to our requests. Therefore, we need to generate this crumb value from the Jenkins CrumbIssuer before making the actual build trigger requests. Here's how we can generate the Jenkins Crumb using python:



def jenkins_crumb():
    session = requests.Session()
    session.auth = ("<JENKINS_USERNAME>", "<JENKINS_USER_PASSWORD>")
    crumb = session.get("https://<YOUR_JENKINS_BASE_URL>/crumbIssuer/api/json")
    return crumb.request.headers["Authorization"], crumb.json()["crumb"]

Step 4: Trigger Jenkins Job from Python

Now, we can finally trigger our job from python scripts. Here's an example,



TOKEN = "<YOUR_AUTH_TOKEN>" # the token value you set in STEP-1

def deploy_service(
    service_name,
    playbook,
    branch="master",
    repo="ashiqursuperfly/jenkins-job-config",
    chart_repo="ashiqursuperfly-portfolio-dev"
):
    JOB_URL = f"https://<YOUR_JENKINS_BASE_URL>/job/<YOUR_JENKINS_JOBNAME>/buildWithParameters?token={TOKEN}"
    url = f"{JOB_URL}&PLAYBOOK={playbook}&GIT_BRANCH_NAME={branch}&GIT_REPO_ID={repo}&HELM_REPOSITORY={chart_repo}"

    cookie, crumb = jenkins_crumb()

    headers = {"Jenkins-Crumb": crumb}
    session = requests.Session()
    session.auth = ("<JENKINS_USERNAME>", "<JENKINS_USER_PASSWORD>")

    res = session.get(url=url, headers=headers)

    if res.ok:
        print(f"Started deploy job for service {service_name} with playbook {playbook}")
    else:
        print(
            f"Error deploy job for service {service_name} with playbook {playbook}\n{res.json()}"
        )

if __name__ == "__main__":
    list_of_service = list_services()

    for service in list_of_service:
        res, playbook = get_playbook_name(service)
        deploy_service(service, playbook)
        sleep(100)

Now, we can invoke the following method as many times as we want. In my case, I just looped through my list of services to deploy and invoked this method for each service and just watched Jenkins complete all the tedious work for me as I sipped my coffee, basked in the brightness of my monitor's light, and wondered why life is so simple when you can't stop thinking automation 🤓

If you found this post helpful, CLICK BELOW 👇

EKS multiple Kubernetes version upgrade using Terraform

Ashiqur Rahman — Sun, 04 Jun 2023 12:01:16 +0000

Recently, We found ourselves using Kubernetes 1.22 for our EKS clusters at work. As of June 4, K8s 1.22 will be deprecated for EKS and AWS recommends migrating to atleast 1.23. Since, we seem to be forced to upgrade every few months, we decided we should explore upgrading to K8s 1.26 if possible in one go.
However, there's one big issue we need to solve in order to migrate to 1.26 directly. Clusters managed by EKS can only be upgraded one minor version at a time, so if you are currently at 1.22, you can upgrade to 1.23. If you are on a lower version like 1.20, you could upgrade to 1.21, then 1.22, then 1.23.

In this post, I'll discuss how we attempted to gracefully navigate this problem.

First of all, we manage our EKS cluster configuration using Terraform.
Our terraform configuration for EKS module looks sth like this,

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "19.5.1"

  cluster_name    = local.cluster_name
  cluster_version = "1.21"

  vpc_id                         = module.vpc.vpc_id
  subnet_ids                     = module.vpc.private_subnets
  cluster_endpoint_public_access = true

  eks_managed_node_groups = {
    one = {
      name = "node-group-1"

      asg_desired_capacity      = 2
      asg_min_size              = 1
      asg_max_size              = 4
      instance_type             = m6a.large
      ami_id                    = "Select EKS-Optimized-AMI-here: https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html"
    }

    two = {
      name = "node-group-2"

      asg_desired_capacity      = 2
      asg_min_size              = 1
      asg_max_size              = 4
      instance_type             = m6a.xlarge
      ami_id                    = "Select EKS-Optimized-AMI-here: https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html"
    }
}

Typically the upgrade steps look something like this,

Update the version and linux_ami to the version you want to upgrade

a. linux_ami for the version can be found here. Choose the k8s version and x86 AMI.
b. Apply the changes in terraform, this can take 1hour or longer.

Run an instance refresh for the autoscaling group the k8s belongs to. You can start the instance refresh either from AWS console or if you would want to automate this step for each of the ASGs. You can do sth like this,

for line in $(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names | grep AutoScalingGroupName | sed 's/\\"AutoScalingGroupName\\": //' | sed -E 's/^\\s+//' | sed -E 's/\\"//' | sed -E 's/\\"//' | sed -E 's/\\,//' | grep spot-group); do
aws autoscaling start-instance-refresh \\
--auto-scaling-group-name $line \\
--preferences '{"MinHealthyPercentage": 99, "InstanceWarmup": 300, "SkipMatching": false}'
done

Upgrade CoreDNS, Kubeproxy, Amazon VPC CNI and other related deployments
Upgrade Helm (Helm Releases)
Upgrade Kubectl (Kubectl Version Policy) and python Kubernetes client (Python Kubernetes Compatibility)

For ASGs that have a large number of nodes the instance refresh takes quite long therefore, that's the step that takes the longest time. Furthermore, we run our instance refreshes with "MinHealthyPercentage": 99 to ensure zero to minimal downtime during the upgrade. This comes at the cost of the instance refresh taking even longer.

However, as long as the instance refresh is not running the autoscaling groups will not replace the older version nodes which means your deployments will keep on working perfectly in between the cluster version upgrades.

Which leads us to the solution,

If we have to make multiple version jumps we can do it quite quickly by running the instance refresh only once after upgrading the cluster to latest version.

Here's a sample of what the process might look like:

Upgrade to K8s 1.23 and update node AMI. Apply using Terraform.
Upgrade to K8s 1.24 and update node AMI. Apply using Terraform.
Repeat untill you reach desired version
Run Instance Refresh
Upgrade kubeproxy, coredns and other related deployments/add-ons to the latest compatible versions
Upgrade Helm, Kubectl and python kubernetes client to latest compatible versions

Voila!

If you found this post helpful, CLICK BELOW 👇

Python script to find unused EC2 subnets

Ashiqur Rahman — Sun, 04 Jun 2023 10:57:47 +0000

Recently, I was working on managing all VPC related resources using Terraform for all our AWS accounts. For one of the accounts, I stumbled upon a rather unpleasant situation where I saw a few hundred subnets in a region and I was left wondering whether all these subnets are actually in use.

Since, I am too lazy to go through them one by one manually, I wrote a python script to check which of these subnets are actually serving a purpose i.e has an ENI attached to it.

Step 1: List all subnets in our VPC

import subprocess
import json

def list_subnets(vpc_id):
    subnets = json.loads(subprocess.getoutput('aws ec2 describe-subnets'))['Subnets']

    def _filter(subnet):
        if subnet['VpcId'] == vpc_id:
            return True
        return False

    def _map(subnet):
        tags = subnet['Tags']
        name = ''
        for tag in tags:
            if tag['Key'] == 'Name':
                name = tag['Value']
        return {
            subnet['SubnetId']: (name, subnet['CidrBlock'], subnet['AvailabilityZone'], subnet['MapPublicIpOnLaunch'])
        }

    return list(map(_map, filter(_filter,subnets)))

Step 2: Check if a subnet has an ENI attached

def check_if_subnet_has_enis_attached(subnet_id):
    cmd = f"aws ec2 describe-network-interfaces --filters Name=subnet-id,Values={subnet_id} --query 'NetworkInterfaces[*].Description'"
    enis = list(json.loads(subprocess.getoutput(cmd)))

    for eni in enis:
        if len(str(eni).strip()) == 0:
            enis.remove(eni)

    return enis

Step 3: Find list of all unused subnets

if __name__ == '__main__':
    subnets = list_subnets()

    unused = []
    for subnet in subnets:
        for k, v in subnet.items():
            if len(check_if_subnet_has_enis_attached(k)) == 0:
                unused.append(subnet)

    print(unused)

If you found this post helpful, CLICK BELOW 👇

This AWS CLI command can run instance refresh for auto scaling groups

Ashiqur Rahman — Wed, 31 May 2023 21:49:39 +0000

Recently, I was working on upgrading Kubernetes versions for all our EKS clusters. The upgrade process is simple, We upgrade the K8s version and instance AMIs from Terraform. Then, we upgrade chart/image version of the different tools we deploy e.g: CoreDNS, ExternalDNS, kubernetes-dashboard, kube-proxy etc.
All these steps can be automated ofcourse since we use Terraform. Especially when you have to upgrade a large number of clusters automation becomes absolutely essential. One step that always bothered me is that, you need to run an instance refresh for the autoscaling groups that your K8s worker nodes belong to. Otherwise, your EKS cluster will still be using the older worker nodes (using the older K8s version). Previously, I always did this using the AWS console manually. So, this was always a unwanted manual step in this workflow.
Recently, I discovered this AWS CLI command that turns out to be exactly what i was looking for.

First you can fetch the autoscaling groups in your account



aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names | grep AutoScalingGroupName | sed 's/\"AutoScalingGroupName\": //' | sed -E 's/^\s+//' | sed -E 's/\"//' | sed -E 's/\"//' | sed -E 's/\,//' | grep worker

Now, we can run the instance refresh for each item in the list using the following command



for line in $(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names | grep AutoScalingGroupName | sed 's/\"AutoScalingGroupName\": //' | sed -E 's/^\s+//' | sed -E 's/\"//' | sed -E 's/\"//' | sed -E 's/\,//' | grep spot-group); do
  aws autoscaling start-instance-refresh \
    --auto-scaling-group-name $line \
    --preferences '{"MinHealthyPercentage": 99, "InstanceWarmup": 300, "SkipMatching": false}'
done

If you found this post helpful, CLICK BELOW 👇

Terraform Unimport : The Terraform command that does not exist !

Ashiqur Rahman — Thu, 25 May 2023 20:23:56 +0000

Terraform import is a great feature to import existing resources into terraform state. But, what if we accidentally imported some resource into terraform state and we want to unimport them again.
Well, terraform does not really have a command/feature that does exactly that but here's one cool trick you can use to quickly unimport some resources from Terraform state.

terraform state rm

While running the terraform command to remove a resource from the state, make sure you quote(') the resource id appropriately like so,

terraform state rm 'module.s3.module.s3_bucket["my-s3-bucket-name"].aws_s3_bucket.this[0]'

Terraform state list can often be very long. If you need to remove a group of resources sharing a common name you can use the following bash oneliner,

terraform state rm $(terraform state list | grep production-files)

For example, the command above will remove a aws_s3_bucket, aws_s3_bucket_policy, aws_s3_bucket_cors_configuration and so on.

Bonus

Utilize the -dry-run flag before performing a potentially destructive operation like this.

terraform state rm 'resource_id' -dry-run

Output:

Acquiring state lock. This may take a few moments...
Would remove module.s3.module.s3_bucket["production-files"].aws_s3_bucket.this[0]
Releasing state lock. This may take a few moments...

Happy Terraforming !

Note: This SO answer i posted recently inspired me to share this here !

If you found this post helpful, CLICK BELOW 👇

Quickest way to make vanilla React.js app SEO friendly

Ashiqur Rahman — Sat, 23 Jul 2022 19:00:46 +0000

Have you ever been put up in a situation where you have to make a huge React.js app SEO-friendly but you dont have much time in your hand to migrate the app to a framework like Next.js or Gatsby.js that supports server-side-rendering?
This solution im gonna discuss today can really help you server-side-render your react application in just a few minutes using a tool like Selenium and a webserver.

NOTE: There seems to be a common misinformation on the internet that the library React-Helmet is the SEO solution for React. But, ITS NOT, atleast not all by it's own. Its important to understand that, React Helmet uses javascript to insert the <meta> tags into the DOM. However, when the google or facebook bot comes to crawl your website, they don't execute javascript. Therefore, the page the bots see when they come to your website does not contain the <meta> tags and the bots can't learn much about your site. Another library that works the same way as React-Helmet is React-Meta-Tags. We would still need a library like this but that will only work after we implement the ideas discussed farther down this post.

In my case, the REST API that the React front end was consuming was built using python. So, Im gonna be using the python Selenium package. But, you can use the idea regardless of which backend technology your project uses. Another thing i want to mention is that, my React app was being served by a Nginx webserver. But again, you should be able to apply the idea which basically just requires you to update the config of whatever webserver you are using.

Solution

Step 1: Update React App webserver config

As mentioned earlier, the React app i was working on was being served through Nginx. Here's what i changed in the existing Nginx config,

 location / {
     set $server_side_render 0;
     set $server_side_render_host_path api.mydomain.com/v1/ssr/;
     if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp") {
         set $server_side_render 1;
     }
     if ($uri !~ "^(.*)/(product|category|other_endpoints_i_want_to_be_seo_friendly)(.*)"){
         set $server_side_render 0;
     }
     if ($server_side_render = 1) {
         proxy_pass https://$server_side_render_host_path$request_uri;
     }
     try_files $uri $uri/ /index.html;
}

The idea behind the change is that, we want to check when one of the bots of a popular site like Facebook or Google comes to our site, and then delegate those requests to a specific endpoint on our backend API. We are calling this endpoint api.mydomain.com/v1/ssr/. You might be wondering why just send the bots to this endpoint? Why not send everyone? I would not recommend doing it because, obviously it would be super slow for an actual user of your website to go through all this just to receive a response from your site. Luckily, google bot and the other bots have a long enough timeout, so its still fast enough for the bots but not as fast for the real users. If you want to serve server-side-rendered html to all of your users, you should consider migrating to a framework like Next.js or Gatsby.js. But, thats gonna take a good amount of time too if your React app is large enough which is exactly why i think the aprroach i am discussing in this post is relevant.

Step 2: Add backend API /ssr/ endpoint

Now, that we have sent the bots to this endpoint, we need to serve them javascript rendered html files for their request_uri. This is where Selenium comes in, we can use it just to render html on the backend. Here's how it works,

def render_html(request, path):
    if request.method != 'GET':
        return HttpResponse(status=405)

    # start after 3 letters (s-s-r)
    idx = request.get_full_path().find('ssr/')
    url = f"https://www.mydomain.com{request.get_full_path()[idx + 3:]}"

    chrome_options = Options()
    chrome_options.add_argument("--disable-extensions")
    chrome_options.add_argument("--enable-javascript")
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument("--disable-dev-shm-usage")
    chrome_options.add_argument('--ignore-ssl-errors=yes')
    chrome_options.add_argument('--ignore-certificate-errors')
    chrome_options.add_argument('--disable-web-security')
    chrome_options.add_argument('--enable-logging=stderr --v=1')
    # chrome_options.add_experimental_option('w3c', False)
    chrome_options.add_argument('user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36')

    d = DesiredCapabilities.CHROME
    d['goog:loggingPrefs'] = {'browser': 'ALL'}
    driver = webdriver.Chrome(chrome_options=chrome_options, desired_capabilities=d)

    driver.get(url)

    soup = BeautifulSoup(driver.page_source, 'html.parser')

    for entry in driver.get_log('browser'):
        print('Selenium-Log:', entry)

    meta = soup.html.findAll('meta')
    for item in meta:
        print(item)

    driver.quit()
    return HttpResponse(soup.html, content_type="text/html")

We use chrome webdriver and use the option --enable-javascript to find a javascript rendered html string of the website. This html string will contain the appropriate<meta> tags added by libraries like React-Helmet. Thus, we are sending a server-side-rendered html to the bots that come to our site.

Step 3: Add appropriate tags to the react pages

Now, we can use a library like React-Helmet or React-Meta-Tags to inject the tags for each page.

Step 4: Testing

We can test if the system we designed is working using a tool like Facebook-Sharing-Debugger to check what the facebook bot sees when it hits one of the SSR enabled endpoints in our website.

Voila ! We have successfully tricked the bots into seeing a server-side-rendered html of our site which contains the appropriate <meta> tags used for SEO and SMO.

BTW, consider caching these server-side-rendered htmls to make the bots 🤖 🤖 🤖 even happier xD

Jenkins controller-agent (master-slave) setup in 10 minutes using Docker

Ashiqur Rahman — Sat, 23 Jul 2022 09:16:00 +0000

Automation lies at the heart of DevOps. Various automation tools and techniques have truly enabled the concept of continuous integration and continuous delivery. These tools have evolved over the years rapidly but one name that seems to be here forever is Jenkins.

We will neither discuss introductory concepts of CI-CD in this post nor waste time showing Jenkins installation steps. If you are new to Jenkins, you can check out the Offical Installation Docs to get started with Jenkins. Therefore, the purpose of this post is to discuss how to setup Jenkins controller-agent architecture (aka master-slave architecture) and address some of the issues that arise when doing it. This is because, the process can be tedious and if you haven't done it in a while, you may end up wasting a few hours.

Why Jenkins Controller-Agent Architecture?

The controller(master) node is Jenkin's brain, it's where the Jenkins application runs. If we do too much work on the controller node (or it crashes), the entire application may become unavailable. Therefore, we want the master to be as available as possible. This can be done by delegating work to agent nodes (slave nodes). So in a Jenkins Controller-Agent architecture, the jobs are scheduled and assigned to agents by the controller. The controller also keeps track of whether the slaves are online, retrieve their responses of the build results, and outputs the build results on the console. Thus, the master node is more available and hence the overall performance of our Jenkins server is improved using this design.

Another advantage of this architecture is that we can only install a minimum set of tools on the controller node and we can install the heavier tools (required by the jobs) on the agent nodes. This keeps the controller lightweight and also allows us to organize our jobs based on the agents that should execute them. In the above example, we have a Jenkins controller along with 4 agents. Each of the agents can serve a specific purpose.

For example, if we need to run tests and build jobs of javascript-based applications we can restrict these jobs to be executed on the agent on the far left.
Similarly, if we need to build some .NET applications we can setup a Jenkins agent with a windows host and restrict these jobs to be executed on the far right.
Furthermore, we can improve performance by balancing the loads based on our system's requirements. Let's say, we are setting up CI-CD of a system that has hundreds of microservices among which, there are twice as many services written using python based stack as any other stack. In this situation, we can setup two Jenkins agents with python based tools installed and the Jenkins controller can balance the load between these two agents.

Setup

Step 1: Spin up a Jenkins Controller(master) Container

We can use the official jenkins docker container. Here's an example docker-compose file that you can use.



version: '3.8'

services:
  jenkins_controller:
    image: jenkins/jenkins:lts-jdk11
    privileged: true
    user: root
    container_name: $CONTAINER_NAME
    ports:
      - 50001:8080
      - 50002:50000
    volumes:
      - $JENKINS_HOME:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock

Now, we need to spin up the container and we need to install the required tools for the Jenkins controller node. We can write a simple bash script to achieve that.



#!/bin/bash
set -o nounset

JENKINS_CONTAINER_NAME=$1
JENKINS_HOME=`pwd`/jenkins/jenkins_home 
echo "JENKINS HOME: $JENKINS_HOME"

CONTAINER_NAME=$JENKINS_CONTAINER_NAME JENKINS_HOME=$JENKINS_HOME docker-compose -f docker-compose-controller.yaml up --build -d

sleep 10

# Here we have just installed git for our controller node, install as many tools as you require
docker exec $JENKINS_CONTAINER_NAME bash -c "apt-get update -y -q && apt-get upgrade -y -q && apt-get install -y -q git"

Jenkins uses apache jetty which uses port 8080 by default, We mapped our host machine's port 50001 with the container's 8080. So, entering http://host:50001 should take you to the Jenkins web dashboard.
Check container logs for the first time admin password and create a new admin user.

Step 2: Set up a Jenkins Agent (slave)

We can now setup our agent(s). Since, our Jenkins controller will communicate with the agents using SSH, we need to generate the SSH keys. In this case, the Jenkins master node will act as the SSH client and the agent(s) will act as SSH servers. So, we need to set it up accordingly.

Generate Keys ```

ssh-keygen -t rsa -f jenkins_agent_1


2. Goto *Jenkins Dashboard* > *Manage Jenkins* > *Manage Credentials* > Add 'System' scoped Credential for enabling SSH into a Jenkins Agent

*System Credential vs Global Credential*
*System*: Only available on Jenkins server (not visible by jenkins job)
*Global*: Accessible everywhere including jenkins job

**Fill up the form with the appropriate values. Here's an example,**
*Username*: jenkins # we want to ssh into the agent as 'jenkins' user which already exists by default in the jenkins-agent container we will be using

*ID*: An Unique ID for the credential that can be used to refer to the credential

*Private Key*: SSH Private Key file contents (e.g: jenkins_agent_1)

#### Step 3: Spin up a Jenkins Agent(slave) Container
We can use the official jenkins-ssh-agent docker container. Here's an example docker-compose file that you can use.
```yaml


version: '3.8'

services:
  jenkins_agent:
    image: jenkins/ssh-agent:jdk11
    privileged: true
    user: root
    container_name: $CONTAINER_NAME
    expose:
      - 22
    environment:
      - JENKINS_AGENT_SSH_PUBKEY=$JENKINS_AGENT_SSH_PUBKEY

Notice that, we must set the environment variable JENKINS_AGENT_SSH_PUBKEY which in this case we are doing from a bash variable. We also need to install the required tools in our Jenkins agent. We can achieve all that using a simple bash script like the one below,



#!/bin/bash

set -o nounset

JENKINS_CONTAINER_NAME=$1

JENKINS_AGENT_SSH_PUBKEY=$2

CONTAINER_NAME=$JENKINS_CONTAINER_NAME JENKINS_AGENT_SSH_PUBKEY=$JENKINS_AGENT_SSH_PUBKEY docker-compose -f docker-compose-agent.yaml up --build -d

sleep 10

# Here we have just installed tools that help us create a python virtual environment for our agent node, install as many tools as you require

docker exec $JENKINS_CONTAINER_NAME bash -c "apt-get update -y -q && apt-get upgrade -y -q && apt-get install -y -q git python3 python3-venv"

Step 4: Configure an Agent from the Jenkins Controller

Goto Jenkins Dashboard > Manage Jenkins > Manage Nodes and Clouds > New Node

Fill up the form using the appropriate values. Here's an example,
Name: JenkinsAgent1

NumberOfExecutors: 1-2

RemoteRootDirectory: /home/jenkins/agent

Labels: linux, python # Space separated values, Can be useful to restrict jobs to run on a particular agent

Usage: Use this node as much as possible

Launch Method: Launch agents via SSH

Host: jenkins_agent # Agent's Hostname or IP to connect. (docker-compose service name if controller and agent is on the same machine)

Credentials: Select the Credential created in Step 2

HostKeyVerificationStrategy: Non verifying Verification Strategy

Launch Method > Advanced

ConnectionTimeoutInSeconds: 60
MaximumNumberOfRetries: 10
SecondsToWaitBetweenRetries: 15

There are some other options you might wanna look into, but the ones i discussed should help you get started.

We can create as many agents as we require using the process discussed above.

Step 5: Create jobs and run

Now, our agent should be discovered by the controller and we can start delegating our jobs to the agents. We can restrict a job to run on a particular agent by using the labels we assigned when creating an agent.

That's done :D. Let me know, if missed anything. You can also find the source files here: ashiqursuperfly/Jenkins-Controller-Agent-Setup

If you found this post helpful, CLICK BELOW 👇

DEV Community: Ashiqur Rahman

Enforcing Kubernetes Probes with a Custom Admission Webhook

Introduction

Why Enforce Probes?

Design Overview

Demo

Pod in default namespace with no probes

Pod in kube-system namespace with no probes defined

Pod in default namespace with both liveness and readiness probes defined

Installation

Pre-requisites

Implementation

Docs

Handling mapping changes in Elasticsearch and reindexing

Understanding Mappings

Why Mapping Changes Occur

Challenges with Mapping Changes

Steps to Handle Mapping Changes

Best Practices

A guide to debugging a Kubernetes deployment

A Guide to Debugging a Kubernetes Deployment

Table of Contents

Understanding the Basics

Common Issues and Solutions

1. Pods Not Starting

2. Image Pull Failures

3. Configuration Errors

4. Networking Issues

Tools and Commands

Best Practices

Conclusion

How to deploy and manage a RabbitMQ cluster on Amazon EKS using Terraform and Helm

When should you consider deploying RabbitMQ on Kubernetes

You can deploy RabbitMQ on a single cloud VM (e.g: on an EC2 machine)

Use a cloud provider-managed RabbitMQ e.g. AmazonMQ.

Deploy it on Kubernetes!

Challenges with deploying RabbitMQ on a Kubernetes cluster:

How should we persist RabbitMQ state?

How should the different nodes in a RabbitMQ cluster communicate with each other inside Kubernetes?

How should we automate the provisioning of as many RabbitMQ instances as our applications need?

Terraform module to provision a list of RabbitMQ instances

instances.yaml

variables.tf

main.tf

Optimize EC2 cost by scheduling instances using EventBridge scheduler and Lambda

Step 1: Lambda

Step 2: Create EventBridge schedule

Automate Jenkins Job builds with Python

Step 1: Enable Triggering Jenkins jobs from remote scripts

Step 2: Parameterize Job -> GIT SCM (BRANCH, REPO ETC.)

Step 3: Generate Jenkins Crumb

Step 4: Trigger Jenkins Job from Python

EKS multiple Kubernetes version upgrade using Terraform

Typically the upgrade steps look something like this,

Python script to find unused EC2 subnets

Step 1: List all subnets in our VPC

Step 2: Check if a subnet has an ENI attached

Step 3: Find list of all unused subnets

This AWS CLI command can run instance refresh for auto scaling groups

Terraform Unimport : The Terraform command that does not exist !

terraform state rm

Bonus

Quickest way to make vanilla React.js app SEO friendly

Solution

Step 1: Update React App webserver config

Step 2: Add backend API /ssr/ endpoint

Step 3: Add appropriate tags to the react pages

Step 4: Testing

Jenkins controller-agent (master-slave) setup in 10 minutes using Docker

Why Jenkins Controller-Agent Architecture?

Setup

Step 1: Spin up a Jenkins Controller(master) Container

Step 2: Set up a Jenkins Agent (slave)

Step 4: Configure an Agent from the Jenkins Controller

Step 5: Create jobs and run

Pod in `default` namespace with no probes

Pod in `kube-system` namespace with no probes defined

Pod in `default` namespace with both liveness and readiness probes defined