DEV Community: Raju Nandi

Your DevOps Tip#2 - On Managing Terraform Variables

Raju Nandi — Tue, 29 Oct 2024 17:39:52 +0000

Hi and welcome to Your Devops Tip blog series. In this series I am going to share some interesting scenarios, challenge and solutions on topics related to realtime use cases on Cloud and Devops.

Scenario

Imagine you're working with Terraform and managing multiple .tfvars files to define your infrastructure variables.

Challenge

How do you ensure that the correct variable values are applied, especially when dealing with multiple .tfvars files with potentially conflicting settings?

Solution

Understanding how Terraform prioritizes these files can streamline your configuration management and avoid unexpected results.Here’s a quick glance on how Terraform handles variable files:

Automatic .tfvars Files

abc.auto.tfvars and terraform.tfvars are picked up automatically by Terraform.
abc.auto.tfvars has precedence over terraform.tfvars.
Example:

abc.auto.tfvars has below values
instance_type = "t3.medium"
terraform.tfvars has below values
instance_type = "t2.micro"

Result: Terraform will use instance_type = "t3.medium" during terraform plan/apply.

Custom .tfvars Files

Custom files (e.g., xyz.tfvars) require you to explicitly specify them using the -var-file option. These files override all other .tfvars settings when provided.

Example

xyz.tfvars has below value instance_type = "p3dn.24xlarge"

When specified during terraform plan or terraform apply, instance_type = "p3dn.24xlarge" will be used. If xyz.tfvars does not include all variables, Terraform will use values from terraform.tfvars or abc.auto.tfvars for any missing variables.

Now you know this, you can control which variables are applied and manage multiple .tfvars files effectively in Terraform.
The solution provided above is focussing on using native terraform techniques. There are wrappers like Teragrunt is also available for handling such scenarios.

Hope this helps streamline your Terraform configurations!

For more techbytes on Cloud and Devops please stay connected to NandiTechBytes on Youtube. Also I am sharing the playlist here for you , please check it out.

Playlist Cloud & Devops Projects and Tasks

Cheers
Keep Learning!
@NandiTechBytes

Your DevOps Tip#1 - On AWS Networking and EKS

Raju Nandi — Fri, 25 Oct 2024 01:30:00 +0000

Hi and welcome to Your Devops Tip blog series. In this series I am going to share some interesting scenarios, challenge and solutions on topics related to realtime use cases on Cloud and Devops.

Scenario

Imagine you have two EKS clusters in separate VPCs with no direct network connectivity (like Transit Gateway or VPC peering) between the two services and you are not allowed to make a connection using TGW or VPC peering between the two VPC’s.

Challenge

There is requirement that an application from VPC-A EKS cluster wants to connect a service on VPC-B EKS cluster. How can you make that happen?

Solution

By Using hashtag AWS PrivateLink , NLB, VPC Endpoints. Here is a brief explanation of the steps.

Create a Network Load Balancer (NLB) for the microservice in VPC B.
Create a VPC Endpoint Service in VPC B and Register the NLB as a VPC Endpoint Service. Modify the microservice in VPC B to be accessible via the NLB.
Create an Interface VPC Endpoint in VPC A that connects to the VPC Endpoint Service in VPC B. This will create an endpoint network interface in VPC A, providing a private IP address to access the service in VPC B.
Update Security Groups:
- Ensure the security groups in VPC B (associated with the NLB and the microservice) allow inbound traffic from the VPC Endpoint in VPC A.
- Similarly, configure the security groups in VPC A to allow outbound traffic to the VPC Endpoint.
Modify the Microservice in VPC A to Use the VPC Endpoint

By following these steps, you can enable secure, private communication between microservices in different EKS clusters without exposing them to the internet or requiring direct network connectivity between the VPCs.

For more techbytes on Cloud and Devops please stay connected to NandiTechBytes on Youtube. Also I am sharing the playlist here for you , please check it out.

Playlist Cloud & Devops Projects and Tasks

Cheers
Keep Learning!
@NandiTechBytes

How AWS SSM agent communicates with 𝐒𝐲𝐬𝐭𝐞𝐦 𝐌𝐚𝐧𝐚𝐠𝐞𝐫 𝐒𝐞𝐫𝐯𝐢𝐜𝐞 using 𝐕𝐏𝐂 𝐈𝐧𝐭𝐞𝐫𝐟𝐚𝐜𝐞 𝐞𝐧𝐝𝐩𝐨𝐢𝐧𝐭𝐬.

Raju Nandi — Wed, 23 Oct 2024 01:04:00 +0000

If you are reading this blog then you are interested to know how the SSM agent running on the EC2 instance is communicating with the AWS System Manager Service.

Basically the SSM agent needs internet access to reach to the System Manager Service but what if your instances are in a restricted environment where they are not allowed to access the internet then how can you use AWS System Manager for managing your instances.

Curious to know how you can use all the features of AWS System Manager inspite of running your instances on a restricted environment. Watch this video
SSM on EC2 with No Internet? Here's How!

Below is how the communication happens from SSM agent running on EC2 to AWS System Manager.

1️⃣ 𝐂𝐚𝐥𝐥𝐬 𝐈𝐧𝐬𝐭𝐚𝐧𝐜𝐞 𝐌𝐞𝐭𝐚𝐝𝐚𝐭𝐚 𝐒𝐞𝐫𝐯𝐢𝐜𝐞: The SSM agent gets the instance metadata for example AWS region.
2️⃣ 𝐃𝐍𝐒 𝐋𝐨𝐨𝐤𝐮𝐩 𝐟𝐨𝐫 𝐀𝐏𝐈 𝐄𝐧𝐝𝐩𝐨𝐢𝐧𝐭: The SSM agent attempts to resolve the API endpoint (e.g., ssm..amazonaws.com) via the private DNS.
3️⃣ 𝐏𝐫𝐢𝐯𝐚𝐭𝐞 𝐃𝐍𝐒 𝐑𝐞𝐬𝐨𝐥𝐯𝐞𝐬 𝐭𝐨 𝐕𝐏𝐂 𝐄𝐧𝐝𝐩𝐨𝐢𝐧𝐭: The private DNS resolves the SSM API domain to the private IP address of the VPC interface endpoint’s ENI.
4️⃣ 𝐓𝐫𝐚𝐟𝐟𝐢𝐜 𝐑𝐨𝐮𝐭𝐞𝐝 𝐭𝐨 𝐄𝐧𝐝𝐩𝐨𝐢𝐧𝐭 𝐄𝐍𝐈: The EC2 instance sends the API request to the private IP address of the VPC interface endpoint's ENI.
5️⃣ 𝐏𝐫𝐢𝐯𝐚𝐭𝐞𝐋𝐢𝐧𝐤 𝐂𝐨𝐦𝐦𝐮𝐧𝐢𝐜𝐚𝐭𝐢𝐨𝐧: The VPC interface endpoint forwards the request over AWS PrivateLink to the AWS SSM service.
6️⃣ 𝐒𝐒𝐌 𝐒𝐞𝐫𝐯𝐢𝐜𝐞 𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐞𝐬 𝐑𝐞𝐪𝐮𝐞𝐬𝐭: AWS Systems Manager processes the API request and Response Sent via PrivateLink to the VPC interface endpoint
7️⃣ 𝐑𝐞𝐬𝐩𝐨𝐧𝐬𝐞 𝐃𝐞𝐥𝐢𝐯𝐞𝐫𝐞𝐝 𝐭𝐨 𝐒𝐒𝐌 𝐀𝐠𝐞𝐧𝐭: The VPC interface endpoint forwards the response to the EC2 instance, where the SSM agent receives and processes it.

For more Tech Bytes on Cloud and Devops you can view the below playlist or follow my channel NandiTechBytes.
📽 Devops Projects & Tasks

Cheers
Keep Learning!

In 𝐊𝐮𝐛𝐞𝐫𝐧𝐞𝐭𝐞𝐬 , Do you know the difference between 𝐕𝐨𝐥𝐮𝐦𝐞 𝐁𝐢𝐧𝐝𝐢𝐧𝐠 and 𝐕𝐨𝐥𝐮𝐦𝐞 𝐌𝐨𝐮𝐧𝐭𝐢𝐧𝐠 ?

Raju Nandi — Sun, 20 Oct 2024 13:00:00 +0000

In this blog , I am going to discuss the two familiar terms in kubernetes world.

𝐕𝐨𝐥𝐮𝐦𝐞 𝐁𝐢𝐧𝐝𝐢𝐧𝐠

The PersistentVolume (PV)represents a storage resource in the cluster.
The PersistentVolumeClaim (PVC)is a request for storage.
The binding process occurs when a PVC is associated with a PV that meets the requested storage size and other specifications.
This step is about reserving storage for the pod, but the actual storage is not yet attached to the pod.

𝐕𝐨𝐥𝐮𝐦𝐞 𝐌𝐨𝐮𝐧𝐭𝐢𝐧𝐠

After the binding is complete, the pod will mount the storage.
The mounted PV (from the PVC) is attached to a specific directory in the pod’s filesystem, allowing the pod to read from or write to the PV. The below image illustrates both the process.

For more Tech Bytes on Cloud and Devops you can view the below playlist or follow my channel NandiTechBytes on Youtube.
📽 Devops Projects and Tasks

Cheers
Keep Learning!

Complete CICD with Gitlab Kubernetes Runner, Gitlab KAS Agent, Kaniko and other Devops tools.

Raju Nandi — Sat, 19 Oct 2024 06:17:16 +0000

The format of the blog is like it starts with the scenario, then challenges and finally the solution.

Scenario

A development team is already using a Kubernetes cluster for their applications and now wants to run their CI/CD jobs on the same cluster for better scalability and resource efficiency. However, they face key Challenges

Challenges

𝐒𝐞𝐜𝐮𝐫𝐢𝐭𝐲 𝐑𝐢𝐬𝐤𝐬 𝐰𝐢𝐭𝐡 𝐊𝐮𝐛𝐞𝐂𝐨𝐧𝐟𝐢𝐠: They want to connect GitLab pipelines to Kubernetes without storing KubeConfig files, as it poses a security risk.
𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐃𝐨𝐜𝐤𝐞𝐫 𝐈𝐦𝐚𝐠𝐞𝐬: With Kubernetes moving away from Docker, they need an alternative to Docker-in-Docker (DinD) to build images within the CI/CD pipeline.
𝐌𝐮𝐥𝐭𝐢-𝐄𝐧𝐯𝐢𝐫𝐨𝐧𝐦𝐞𝐧𝐭 𝐃𝐞𝐩𝐥𝐨𝐲𝐦𝐞𝐧𝐭𝐬: They need a simplified way to deploy across dev, test, and prod using a single Helm chart, avoiding complex configurations.
𝐂𝐨𝐝𝐞 𝐐𝐮𝐚𝐥𝐢𝐭𝐲 𝐚𝐧𝐝 𝐒𝐞𝐜𝐮𝐫𝐢𝐭𝐲: Automated tools for code linting and vulnerability scanning are required to maintain high standards across environments.

As a Devops engineer you are tasked to find an integrated solution to overcome these challenges and streamline their CI/CD process.

Solution

In my latest video, I showcase how to implement a complete 𝐂𝐈/𝐂𝐃 pipeline with 𝐆𝐢𝐭𝐋𝐚𝐛 𝐊𝐮𝐛𝐞𝐫𝐧𝐞𝐭𝐞𝐬 𝐑𝐮𝐧𝐧𝐞𝐫𝐬, integrating top DevOps tools like MegaLinter, Kaniko, Trivy, Helm, and the powerful 𝐆𝐢𝐭𝐋𝐚𝐛 𝐊𝐀𝐒 𝐀𝐠𝐞𝐧𝐭.
𝐖𝐡𝐚𝐭’𝐬 𝐢𝐧𝐬𝐢𝐝𝐞?

𝐃𝐨𝐧’𝐭 𝐰𝐚𝐧𝐭 𝐭𝐨 𝐬𝐭𝐨𝐫𝐞 𝐊𝐮𝐛𝐞𝐂𝐨𝐧𝐟𝐢𝐠? 𝐍𝐨 𝐩𝐫𝐨𝐛𝐥𝐞𝐦! See how the 𝐆𝐢𝐭𝐋𝐚𝐛 𝐊𝐀𝐒 𝐀𝐠𝐞𝐧𝐭 connects to Kubernetes clusters.
𝐄𝐟𝐟𝐨𝐫𝐭𝐥𝐞𝐬𝐬 𝐃𝐞𝐩𝐥𝐨𝐲𝐦𝐞𝐧𝐭𝐬 across dev, test, and prod using a single Helm chart.
𝐊𝐮𝐛𝐞𝐫𝐧𝐞𝐭𝐞𝐬 𝐑𝐮𝐧𝐧𝐞𝐫𝐬 running isolated CI/CD jobs in scalable Kubernetes pods.
𝐂𝐨𝐝𝐞 𝐐𝐮𝐚𝐥𝐢𝐭𝐲 𝐀𝐬𝐬𝐮𝐫𝐚𝐧𝐜𝐞 with MegaLinter to keep your codebase clean and compliant.
𝐒𝐞𝐜𝐮𝐫𝐢𝐭𝐲 𝐅𝐢𝐫𝐬𝐭 with Automated vulnerability scanning using Trivy.
𝐁𝐮𝐢𝐥𝐝 𝐚𝐧𝐝 𝐏𝐫𝐨𝐦𝐨𝐭𝐞 with 𝐊𝐚𝐧𝐢𝐤𝐨

I tried to cover everything you need to implement this CI/CD pipeline, making it an absolute must-watch for DevOps engineers and developers alike! 😎
𝐂𝐮𝐫𝐢𝐨𝐮𝐬 𝐡𝐨𝐰 𝐢𝐭 𝐚𝐥𝐥 𝐰𝐨𝐫𝐤𝐬?
Watch the video
🎥 Complete CICD with Gitlab Kubernetes Runner, Gitlab KAS Agent, Kaniko and other Devops tools.

Cheers
Keep Learning