DEV Community: Amash Ansari

Shell Scripting Mini Project

Amash Ansari — Thu, 12 Sep 2024 18:15:00 +0000

About

This project gives detailed information about the server who is the user, date and time, Server Uptime, last login activities, Disk utilization, RAM utilization, and Top CPU Processes Running. This project is made using Shell-Scripting, the script can be executed using the bash. I've also taken a backup of this script in the form of a zip file. Some extra commands are also used just to beautify the project.

Prerequisite

Linux
Shell Scripting

Things to Remember

Before diving into the project, there are a few commands that are required to be discussed just to have a simple understanding of the project.

whoami - allows Linux users to see the currently logged-in user.
echo - is used to display the text passed in as an argument.
echo -e - allows escape sequences within its operands.
date - displays the current time and date (of the system) in the given FORMAT.
xargs - converts the statements into arguments.
awk - a utility that enables a programmer to write tiny but effective programs in the form of statements
top - a commonly used tool for displaying system-performance information.
last - used to display a list of users who have previously logged in to the system.
uptime - an important metric that shows how long the system has been running or how much time has passed since the system last rebooted.
df -h - displays information about total space and available space on a file system.
free -h - displays the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel.

Script

Create a shell script.

 vim server_details.sh

Add production house at the top, which means mention the shell you'll be using for the execution of the script.

 #!/bin/bash

Start writing the script in the way you like by not conflicting with the syntax.
The final script looks something like this. 👇

#!/bin/bash

RED="31"
GREEN="32"
YELLOW="33"
MAGENTA="35"
GRAY="90"
WHITE="97"
ITALICRED="\e[3;${RED}m"
ITALICYELLOW="\e[3;${YELLOW}m"
ITALICWHITE="\e[3;${WHITE}m"
ITALICGREEN="\e[3;${GREEN}m"
BOLDGRAY="\e[1;${GRAY}m"
BOLDYELLOW="\e[1;${YELLOW}m"
BOLDMAGENTA="\e[1;${MAGENTA}m"
ENDCOLOR="\e[0m"


echo -e "${BOLDMAGENTA}\n---------------------------WELCOME----------------------------------\n${ENDCOLOR}"
echo -e "${BOLDYELLOW}\nHey $(whoami), Welcome to the server details corner :-)\n${ENDCOLOR}"

echo -e "${BOLDGRAY}----------------------------***----------------------------------${ENDCOLOR}"
echo -e "${ITALICWHITE}        Current date and Time is : $(date | xargs | awk '{print $3"/"$2"/"$6,$1,$4,$5}')${ENDCOLOR}"
echo -e "${BOLDGRAY}----------------------------***----------------------------------\n${ENDCOLOR}"

echo -e "${BOLDYELLOW}******************************************************************${ENDCOLOR}"
echo -e "${BOLDGRAY}Server Uptime is   : ${ENDCOLOR} ${ITALICWHITE} $(uptime) ${ENDCOLOR}"
echo -e  "${BOLDGRAY}\nLast login details :${ENDCOLOR} ${ITALICWHITE} \n$(last -a | head -5) ${ENDCOLOR}"
echo -e "${BOLDYELLOW}******************************************************************${ENDCOLOR}"

echo -e "${ITALICRED}\n###################################################################${ENDCOLOR}"
echo -e "${ITALICYELLOW}    Disk Space available :${ENDCOLOR} ${ITALICWHITE} $(df -h | xargs | awk '{print $11"/"$9}')${ENDCOLOR}"
echo -e "${BOLDGRAY}\n------------------------------------------------------------------\n${ENDCOLOR}"

echo -e "${ITALICYELLOW}    RAM utilization      :${ENDCOLOR} ${ITALICWHITE} $(free -h | xargs | awk '{print $10"/"$8}')${ENDCOLOR}"
echo -e "${ITALICRED}##################################################################${ENDCOLOR}"

echo -e "${BOLDMAGENTA}\n----------------------------***----------------------------------\n${ENDCOLOR}"
echo -e "${ITALICYELLOW}Top CPU Processes running : ${ENDCOLOR}${ITALICWHITE}"  && top -b | head -10
echo -e "${BOLDMAGENTA}\n----------------------------***---------------------------------\n${ENDCOLOR}"

Note: I've used some variables here just to beautify the project, they're optional to use. "${}" is used for mentioning variables in arguments. If you want to learn more about styling the text in bash script then Click here.

Output 🎉

Scaling Infrastructure Across Environments with Terraform

Amash Ansari — Wed, 11 Sep 2024 17:42:50 +0000

In this post, we'll explore how to set up Multi-Environment Infrastructure through a step-by-step project. We'll use modules to write organized and reusable code, allowing us to create multiple environments using a single setup. We'll enhance the security and reliability of our infrastructure by integrating our Terraform state file, terraform.tfstate with a remote backend (S3 bucket). In addition, we'll implement state locking and log the results in a DynamoDB table.

This post covers advanced Terraform concepts, so it's best if you have a good grasp of Terraform fundamentals and intermediate concepts.

Prerequisite

You should be familiar with AWS services like EC2, S3, and DynamoDB.
It's crucial to have a strong understanding of Terraform basics.
Ensure you have AWS CLI installed on your system. If it's not already installed, you can set it up by clicking here.

If you're new to Terraform basics, you can get started with this beginner-friendly guide by clicking here.

Exploring Project's Implementation

We'll complete this project by writing clean and organized code and keeping related configurations together in a separate folder. This approach makes project understanding and management more efficient and speedy.
We'll make a new folder to store module configurations. In this folder, we'll create modules for EC2, S3, DynamoDB, and a Variable module to manage configuration settings.
We're doing this to write the configurations once and use them to create resources in various environments, such as development, production, and testing. The advantage is that we don't have to write separate configurations for each environment. With modular code, one configuration can create resources for multiple environments.
Last but not least, we'll also create a main file where we'll put our discussed modules to use and create multiple infrastructure environments.

Creating Modules

Start by setting up a dedicated 'modules' folder to house all your modules. Open this folder with your code editor (I'll be using VS Code in this post). Then, create separate files with the .tf extension to define your EC2, S3 bucket, and DynamoDB table configurations.

Choose the variable configuration and begin writing the script as outlined below. This script will include the variables for our modules, which will be utilized in the resource configurations.

  # Variable for multi-environments
  variable "env" {
    description = "This is the environment value for multiple-environments of our infrastructure"
    type        = string
  }

  # Variable for giving ami
  variable "ami" {
    description = "This is the ami value for EC2"
    type        = string
  }

  # Variable for giving instance type
  variable "instance_type" {
    description = "This is the instance_type for our infrastructure"
    type        = string
  }

Now, open the EC2 file and paste the following configuration. This will set up the EC2 instance for multi-environments.

  # EC2 Instance configuration for multi-environments
  resource "aws_instance" "ec2" {
    ami           = var.ami    # Using ami variable
    instance_type = var.instance_type # Using instance_type variable
    tags = {
      Name = "${var.env}-instance" # Using env variable to give instance name
    }
  }

Next, open the S3 file and paste the following configuration. This will create the S3 bucket for multiple environments.

  # S3 bucket configuration for multi-environments
  resource "aws_s3_bucket" "module-bucket" {
    bucket = "${var.env}-amash-bucket" # Using env variable to give bucket name
    tags = {
      Name = "${var.env}-amash-bucket"
    }
  } # NOTE: Bucket name should be unique

Afterward, open the DynamoDB file and insert the following configuration. This will establish the DynamoDB table for various environments.

# DynamoDB configuration for multi-environments
  resource "aws_dynamodb_table" "dynamodb-table" {
    name         = "${var.env}-table" 
    billing_mode = "PAY_PER_REQUEST" # Give billing mode
    hash_key     = "userID"
    attribute {
      name = "userID" # Name of table attribute
      type = "S" # Type of table attribute i.e. string
    }
    tags = {
      Name = "${var.env}-table" # Using env variable to give table name
    }
  }

That's it for the modules. Now, we'll employ these modules in various infrastructure environments.

Creating Main Configurations

Begin by crafting separate .tf files to define your main configurations, such as Terraform settings, providers, and the backend. Here's how it should look:

Next, open the remote-backend.tf file and insert the following configuration. This will link the state file, terraform.tfstate with the remote backend. It will also implement state locking during updates and display logs in a tabular format.

  # Remote backend variable for S3 bucket
  variable "state_bucket_name" {
    default = "demoo-state-bucket"
  }

  # Remote backend variable for DynamoDB table
  variable "state_table_name" {
    default = "demoo-state-table"
  }

  # Variable for giving aws-region
  variable "aws-region" {
    default = "us-east-1"
  }

  # Backend resources for S3 bucket
  resource "aws_s3_bucket" "state_bucket" {
    bucket = var.state_bucket_name 
    tags = {
      Name = var.state_bucket_name # Using state_bucket_name variable to give a bucket name
    }
  }

  resource "aws_dynamodb_table" "state_table" {
    name         = var.state_table_name
    billing_mode = "PAY_PER_REQUEST" # Give any billing mode
    hash_key     = "LockID" # This key will serve as the lock for the infrastructure state

    attribute {
      name = "LockID" # Name of table attribute
      type = "S" # Type of table attribute i.e. string
    }
    tags = {
      Name = var.state_table_name # Using state_table_name variable to give a table name
    }
  }

Select the Terraform configuration and start scripting it as described below. This script defines the high-level behavior of the Terraform configuration.

  # Terraform block
  terraform {
    required_providers {
      aws = {
        source  = "hashicorp/aws"
        version = "~> 5.0"
      }
    }

  # Remote backend configuration
    backend "s3" {
      bucket         = "demoo-state-bucket" # S3 Bucket name
      key            = "terraform.tfstate" # File that we intend to store remotely
      region         = "us-east-1" # Give any region
      dynamodb_table = "demoo-state-table" # DynamoDB table name
    }
  }

After that, open the providers.tf file and insert the following configuration. This is used to specify the region for creating resources.

 # Provider configuration for selecting aws region
  provider "aws" {
    region = var.aws-region # Using aws-region variable from remote-backend file
  }

At last, open the main file and add the following configuration. This file utilizes all the modules we've created so far to set up different environments of our infrastructure, including development, production, and testing.

# Created development environment of infrastructure
  module "dev" {
    source        = "./modules" # Giing path to the modules
    env           = "dev" # Passing the environment variable value, which we've defined in the 'var-module.tf' file within the 'modules' folder
    ami           = "ami-053b0d53c279acc90" # Passing the ami variable value, which we've defined in the 'var-module.tf' file within the 'modules' folder
    instance_type = "t2.micro" # Passing the instance_type variable value, which we've defined in the 'var-module.tf' file within the 'modules' folder
  }

  # Created production environment of infrastructure
  module "prod" {
    source        = "./modules"
    env           = "prod"
    ami           = "ami-053b0d53c279acc90"
    instance_type = "t2.micro"
  }

  # Created testing environment of infrastructure
  module "test" {
    source        = "./modules"
    env           = "test"
    ami           = "ami-053b0d53c279acc90"
    instance_type = "t2.micro"
  }

Our next step is to run these scripts with Terraform and bring our hard work to a successful outcome.

Things to Remember

We're just a few steps away from making this project functional, but before that, there are some important points to grasp. Initially, we'll comment out the remote "s3" {} section in the terraform.tf file and apply the configuration using the terraform apply command. Following that, we'll uncomment the remote "s3" {} part in the terraform.tf file and apply the configuration again using terraform apply. This is necessary due to Terraform's behavior, which runs the backend configuration before any other configuration. Since we've used the S3 bucket and DynamoDB tables in our backend, if the backend uses these resources before their creation, Terraform will generate an error. To avoid this, we'll first create the resources like the S3 bucket and DynamoDB table, and then use them to store the state of our infrastructure.

Terraform in Action

Execute terraform init to initialize the folder with Terraform.

Next, run terraform validate to check the code syntax, and then execute terraform plan to preview the changes that will occur if you apply this configuration.

Finally, perform terraform apply to apply the configuration, and wait for Terraform to provision the multiple environments for you (Don't forget to comment out the remote "s3" {} part in the terraform.tf file).

After the changes have been successfully applied, uncomment the remote "s3" {} part in the terraform.tf file. Remember to execute terraform init first, because we've altered Terraform's behavior by adding a backend. Terraform will need to install the necessary plugins. Then, apply the latest configuration by executing the terraform apply command, as discussed above.

After some time, you'll see all the EC2 instances, S3 buckets, and DynamoDB tables on the AWS console, provisioned with just a single click. With the addition of a remote backend and state locking, this showcases the power of Terraform. You can make infrastructure respond to your commands effortlessly.

Conclusion

We've embarked on a journey to establish Multi-Environment Infrastructure through a comprehensive step-by-step project. By leveraging modules, we've structured our code for reusability, simplifying the creation of multiple environments from a single configuration.

We've taken steps to enhance the security and reliability of our infrastructure by seamlessly integrating our Terraform state file (terraform.tfstate) with a remote backend hosted on an S3 bucket. Moreover, we've prioritized infrastructure stability by implementing state locking and recording logs in a DynamoDB table. This project showcases the power of Terraform, empowering us to manage infrastructure with precision and ease.

Here is the project link🔗 Click to access it.

Effortless Deployment of Project with Ansible & Nginx on AWS

Amash Ansari — Tue, 10 Sep 2024 18:30:24 +0000

Prerequisite

Before you start, you should already know how to write Ansible playbooks well.
You should have some basic knowledge of Nginx.
You should be comfortable with AWS EC2.

If you're new to AWS EC2, check out this beginner-friendly guide. For more information about Ansible, click here.

Create the EC2 Instances

Set up two EC2 instances for this project: one as the control (worker) node, managed by Ansible, and the other as the managed (worker) node. The control node will use playbooks to configure the managed node and automate the entire deployment process.

Let's write the playbook

What we're going to do is create a playbook that installs Nginx and starts the service on the worker node. This playbook will be set up and controlled by the control node. Here's the playbook:

-
  name: This is a simple HTML project
  hosts: servers             #Group name that will host this playbook
  become: yes                #Giving sudo privileges
  tasks:
    - name: nginx-install    #This will install the nginx
      apt:
        name: nginx
        state: latest

    - name: nginx-start
      service:                #This will start and enable the nginx service
        name: nginx
        state: started
        enabled: yes

    - name: deploy-app
      copy:
        src: ../index.html     #Give path to the file
        dest: /var/www/html/   #Nginx will serve our file from this specific location

Our next task is straightforward: run this playbook on the control node using the terminal and wait for Ansible to configure and deploy an end-to-end application with a single click.

If you'd like to access my "index.html" file, just click here.

Execute the playbook

To run the playbook, use the following command:

ansible-playbook <playbook-name>

You'll see some output similar to this:

Check the project by accessing

Finally, ensure the project is up and running by accessing it on the default Nginx port, which is 80.

Grafana 101: A Beginner’s Guide to the Powerful Dashboard Tool

Amash Ansari — Mon, 09 Sep 2024 18:02:33 +0000

Grafana is a powerful tool that helps visualize and monitor data from various sources, making complex information easy to understand through customizable dashboards and graphs.

Grafana's key use cases include monitoring systems, analyzing data, and creating visualizations. Its benefits lie in providing real-time insights, simplifying complex data into easy-to-understand visuals, and aiding in decision-making by spotting trends or issues quickly.

Important Terminologies

Before diving into Grafana, it's important to know some key terms like Observability, which helps understand systems, Monitoring checks how things are doing, Logging for keeping records, and Alerting, which warns about problems.

Monitoring in Grafana involves keeping an eye on systems or data in real time. It helps track how things are performing, like checking if a website is running smoothly or how much CPU a server is using.
Logging in Grafana is like keeping a diary for a computer system. It records events, errors, or important actions that happen, helping to understand what's been going on and troubleshoot any issues later on.
Alerting in Grafana is like having a watchdog. It's there to notify you if something goes wrong or needs attention in your system. It sends warnings or messages when specific conditions you set are met, so you can take action quickly.
Observability in Grafana refers to the ability to understand what's happening within a system through monitoring, logging, and tracing. It involves collecting data and using visualizations to gain insights into system performance, health, and behavior to effectively troubleshoot and optimize it.

Grafana's Inside Story

Graphite and agent?

Graphite is a tool used to store and graph time-series data. A graphite agent (like Carbon) collects data and sends it to Graphite for storage. Grafana, on the other hand, is a visualization tool that can pull data from Graphite and create visuals like charts or graphs.

The graphite-agent collects data from different sources, like servers or applications, and sends it to Graphite. This data might be about server performance, website traffic, or any other metric you want to track. Graphite then stores this data in a way that Grafana can understand.

To display this data in Grafana, you connect Grafana to your Graphite database. Grafana queries Graphite for the data and creates visual representations, such as graphs or charts, allowing you to monitor and analyze the information. Essentially, Graphite acts as the storage and the graphite-agent helps gather data, while Grafana showcases this data visually for easy understanding and analysis.

Loki and Promtail?

In Grafana, Loki is a system used for logging and storing logs. It's designed to handle huge amounts of log data efficiently. Promtail, on the other hand, is like a collector - it gathers logs from different sources (like files or applications) and sends them to Loki for storage.

Think of Loki as a big library where all the logs are stored, and Promtail is like a librarian that collects logs from different places and organizes them neatly in that library so you can easily find and use them later on. This helps in analyzing and troubleshooting issues by having all the logs in one accessible place.

How does Loki and Promtail differ from Graphite?

Loki and Promtail: They're focused on handling and storing logs, like text-based records of events or activities, making it easier to search through and analyze this information.

Graphite: It's more about storing and visualizing numeric, time-based data, like performance metrics or sensor readings, in graphs or charts.

Loki and Promtail are for logs, while Graphite is for numeric data and graphs. They handle different types of information.

Grafana's Editions

Grafana Cloud: It's a managed service offered by Grafana Labs that provides hosting for Grafana, Prometheus, and other related tools. It's like a ready-to-use platform where you can access Grafana without managing servers or infrastructure yourself. It's convenient for quick setup and management.
Grafana Enterprise: This is a version of Grafana designed for larger organizations or teams with more advanced needs. It offers additional features, support, and customization options compared to the open-source version. It's like a souped-up version of Grafana with extra capabilities tailored for bigger setups.

Prometheus

Prometheus is a tool used for monitoring and alerting computing systems. It collects data from various sources, like servers or applications, to help you understand their performance.

Think of it as a health tracker for your systems—it monitors things like CPU usage, memory, or website response times. This data helps spot issues early and sets off alarms when things aren't working as they should, allowing you to fix problems before they become bigger. Overall, it helps maintain system health and performance.

An introduction to AWS Lambda for beginners

Amash Ansari — Sun, 08 Sep 2024 17:58:49 +0000

Understanding AWS Lambda

In the realm of cloud computing, AWS Lambda emerges as a revolutionary service offered by Amazon Web Services (AWS). But what exactly is Lambda? It's a serverless computing platform that allows you to run code without managing servers. Think of Lambda as your assistant, ready to execute your code whenever needed, eliminating the complexities of server setup and maintenance.

The Inner Workings of AWS Lambda

Imagine Lambda as a responsive friend awaiting your call. You upload your code to Lambda and specify triggers that prompt its execution. These triggers could be events like file uploads, database modifications, or even scheduled time intervals. When triggered, Lambda instantly springs into action, executes your code, and then returns to its restful state. It's an efficient system that operates on a pay-as-you-go basis, charging only for the time your code runs.

Advantages of AWS Lambda

Let's unwrap the benefits that make Lambda an attractive choice:

Serverless Simplicity: No more server management headaches. Lambda handles the infrastructure, allowing you to focus solely on crafting your code.
Cost-Effective: Pay only for your code's compute time. There are no charges for idle server time, making it a cost-efficient solution.
Automatic Scaling: Whether your application has ten users or a million, Lambda automatically scales to meet demand without manual intervention.

Applications of AWS Lambda

The versatility of Lambda extends across various domains:

Event-Driven Actions: Use Lambda to respond to incoming messages, image uploads, or database modifications. The code executes instantly upon event occurrence.
Backend Services: Develop APIs or backend logic without the hassle of managing servers. Lambda simplifies the process by handling the infrastructure complexities.
Data Processing: Process and transform data with ease. Lambda's capabilities allow for seamless data handling and computation.

Initiating Your AWS Lambda Journey

Getting started with Lambda is an exciting endeavor:

AWS Account Setup: Begin by signing up for an AWS account if you haven't already.
Navigating to Lambda: Access the AWS Management Console, select "Services," and click on "Lambda" under the "Compute" section.
Creating a Function: Dive into Lambda by creating your first function. Provide a name, select a runtime environment (like Node.js, Python, etc.), and upload your code.
Configuring Triggers: Choose triggers that activate your function, these could be events like file uploads, API calls, or scheduled times.
Testing and Monitoring: Experiment with your function within Lambda, monitor its performance, and refine it as necessary.

Conclusion:

In conclusion, AWS Lambda stands as a game-changer in the landscape of cloud computing. Its ability to streamline code execution without the hassle of server management empowers developers and businesses alike. With its cost-effectiveness, scalability, and diverse applications, Lambda opens doors to a world of seamless computing experiences. Begin your Lambda journey today, and witness firsthand the transformative power of serverless computing!

Setting Up an AWS EKS Cluster Using Terraform: A Beginner-Friendly Guide

Amash Ansari — Sat, 07 Sep 2024 17:47:24 +0000

Amazon Elastic Kubernetes Service (EKS) simplifies running Kubernetes on AWS without having to install or operate your own Kubernetes control plane. In this guide, I’ll walk you through creating an EKS cluster using Terraform — an Infrastructure as Code (IaC) tool that helps automate provisioning.

By the end of this post, you'll have a fully operational Kubernetes cluster running in AWS. Let’s get started!

Prerequisites

Make sure you have the following installed before proceeding:

AWS CLI: To interact with AWS services from your terminal.
Terraform: To manage AWS infrastructure as code.

You also need an AWS account with sufficient permissions to create VPCs, EKS clusters, and security groups.

Step 1: Configure AWS Credentials

First, configure AWS CLI with your access keys by running the following command:

aws configure

Enter your AWS Access Key ID, Secret Access Key, Default region name, and Default output format. Ensure that the IAM user or role you are using has the required permissions to create EKS clusters.

Step 2: Create a VPC Using Terraform

Create a file named vpc.tf to define the network setup. This configuration will create a Virtual Private Cloud (VPC) with both private and public subnets and enable DNS support.

data "aws_availability_zones" "azs" {}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.13.0"

  name = var.vpc_name
  cidr = var.vpc_cidr

  azs             = data.aws_availability_zones.azs.names
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]

  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name                                    = var.vpc_name
    "kubernetes.io/cluster/${var.eks_name}" = "shared"
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${var.eks_name}" = "shared"
    "kubernetes.io/role/internal-elb"       = "1"
  }

  public_subnet_tags = {
    "kubernetes.io/cluster/${var.eks_name}" = "shared"
    "kubernetes.io/role/elb"                = "1"
  }
}

Step 3: Define Variables

Create a file variables.tf to define reusable variables like VPC name, CIDR block, and EKS cluster name. This way, you can easily adjust these values without modifying the core configuration.

variable "aws_region" {
  description = "AWS region"
  default     = "us-east-1"
}

variable "vpc_name" {
  description = "VPC name"
  type        = string
}

variable "vpc_cidr" {
  description = "VPC CIDR"
  default     = "10.0.0.0/16"
}

variable "eks_name" {
  description = "AWS EKS Cluster name"
  type        = string
}

variable "sg_name" {
  description = "Security group name"
  default     = "aws-eks-sg"
}

Step 4: Create Security Groups

Now, let's set up security groups that control network access to the EKS cluster. Create a file security-groups.tf with the following content:

resource "aws_security_group" "eks-sg" {
  name   = var.sg_name
  vpc_id = module.vpc.vpc_id
}

resource "aws_security_group_rule" "eks-sg-ingress" {
  description       = "allow inbound traffic from eks"
  type              = "ingress"
  from_port         = 0
  to_port           = 0
  protocol          = -1
  security_group_id = aws_security_group.eks-sg.id
  cidr_blocks       = ["49.43.153.70/32"]
}

resource "aws_security_group_rule" "eks-sg-egress" {
  description       = "allow outbound traffic to eks"
  type              = "egress"
  from_port         = 0
  to_port           = 0
  protocol          = -1
  security_group_id = aws_security_group.eks-sg.id
  cidr_blocks       = ["0.0.0.0/0"]
}

Step 5: Set Up the EKS Cluster

With the network and security setup complete, create a file eks.tf to define the EKS cluster and node groups.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = var.eks_name
  cluster_version = "1.30"

  enable_irsa = true

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  tags = {
    cluster = "my-eks-cluster"
  }

  # EKS Managed Node Group(s)
  eks_managed_node_group_defaults = {
    ami_type               = "AL2_x86_64"
    instance_types         = ["t2.micro"]
    vpc_security_group_ids = [aws_security_group.eks-sg.id]
  }

  eks_managed_node_groups = {
    node_group = {
      min_size     = 2
      max_size     = 3
      desired_size = 2
    }
  }
}

Step 6: Create Terraform and Provider Files

You also need the following Terraform and provider files for proper setup:

`terraform.tf`

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

`provider.tf`

provider "aws" {
  region = var.aws_region
}

`output.tf`

output "cluster_id" {
  description = "AWS EKS Cluster ID"
  value       = module.eks.cluster_id
}

output "cluster_endpoint" {
  description = "AWS EKS Cluster Endpoint"
  value       = module.eks.cluster_endpoint
}

output "cluster_security_group_id" {
  description = "Security group ID of the control plane in the cluster"
  value       = module.eks.cluster_security_group_id
}

output "region" {
  description = "AWS region"
  value       = var.aws_region
}

output "oidc_provider_arn" {
  value = module.eks.oidc_provider_arn
}

Step 7: Initialize Terraform

Run the following command to initialize Terraform and download the necessary providers:

terraform init

Step 8: Validate and Plan

Before applying the configuration, validate it and preview the changes:

terraform validate
terraform plan

Step 9: Apply the Terraform Configuration

Create your EKS cluster and the associated VPC with:

terraform apply

This process will take approximately 15 minutes to complete. Once done, you’ll have your AWS EKS cluster up and running.

Step 10: Verify the Cluster in AWS Console

Head to the AWS Management Console, navigate to EKS, and confirm that your cluster is listed.

Step 11: Configure Cluster Access (Manual Step)

After the EKS cluster is ready, you need to configure access for your IAM users or roles:

In the AWS Console, navigate to EKS > Your Cluster > Configuration > Access.
Click on Add role or Add user.
Provide the IAM Principal ARN.
Choose the Type (Role or User), provide a Username, and select a Policy Name (e.g., Admin or ViewOnly).
Define the Access Scope.
Click Add Policy and finalize the configuration.

This step grants users or roles permissions to interact with the Kubernetes cluster using IAM-based authentication.

Step 12: Verify Nodes in the Compute Tab

To check if the nodes are up and running:

Go to the Compute tab of the cluster in the AWS Console.
Ensure that the worker nodes are visible.

Step 13: Access the Cluster via CLI

Once the cluster is set up, access it using the AWS CLI by running:

aws eks update-kubeconfig --name <cluster-name> --region <aws-region>

This command configures your kubectl context to use the new EKS cluster.

Step 14: Clean Up Resources

To avoid unexpected charges, delete the resources you created when they’re no longer needed:

terraform destroy

Conclusion

You’ve successfully set up an AWS EKS cluster using Terraform! This guide provides a simplified approach to provisioning cloud infrastructure, ensuring that you can quickly get started with Kubernetes on AWS.

Feel free to customize the configuration as per your needs, and remember to clean up any resources when you’re done to avoid unnecessary costs. Happy Terraforming!

A Beginner's Guide to DevOps: Everything You Need to Know

Amash Ansari — Fri, 06 Sep 2024 17:38:18 +0000

If you're new to the world of software development or IT, you may have heard the term DevOps being thrown around. But what is it, really? Why is everyone talking about it, and how does it transform the way we build, deploy, and maintain software?

In this post, we'll break down the essentials of DevOps and guide you through the following:

What is DevOps?
Why is DevOps important?
Pros of adopting DevOps.
How to get started with DevOps (Roadmap).
Best practices in DevOps.
Who can learn DevOps?
Core skills needed in DevOps.
An overview of DevSecOps and MLOps.

What is DevOps?

At its core, DevOps is a set of practices and cultural philosophies that bring together development (Dev) and operations (Ops) teams to work more efficiently and collaboratively. The goal is to automate and streamline the software development lifecycle (SDLC), from coding to deployment and monitoring, fostering a culture of continuous integration (CI) and continuous delivery (CD).

In simpler terms, DevOps is about breaking down silos between teams, automating processes, and ensuring faster and more reliable software delivery.

Why DevOps?

The traditional software development process often involved separate teams for coding, testing, and deploying. This could lead to long delays, miscommunications, and a lack of accountability when issues arose. DevOps solves these problems by unifying the development and operations teams, speeding up the delivery process and improving overall quality.

Here are some reasons why organizations adopt DevOps:

Faster Time to Market: By automating tasks and improving communication, DevOps helps deliver new features and fixes faster.
Improved Collaboration: DevOps promotes a culture of shared responsibility, making teams more cohesive and efficient.
Better Quality: Through continuous integration and testing, DevOps ensures software is regularly tested and improved, leading to fewer bugs and issues.
Higher Stability: With monitoring and feedback loops in place, issues are identified and fixed faster, resulting in more stable systems.

Pros of DevOps

Adopting DevOps offers a wide range of benefits for both organizations and teams:

Increased Efficiency: Automated pipelines reduce manual tasks, freeing up developers and operations staff to focus on high-value work.
Reduced Costs: By streamlining processes, DevOps reduces unnecessary overhead and errors, lowering operational costs.
Enhanced Security: Through practices like Infrastructure as Code (IaC) and automated security testing, security becomes integrated into the development pipeline.
Faster Recovery: In case of failure, DevOps practices allow for quicker issue detection and recovery.
Scalability: DevOps encourages practices that make scaling infrastructure and applications seamless.

How to DevOps: A Roadmap for Beginners

Ready to dive into DevOps? Here’s a simple roadmap to guide your journey:

Learn Version Control: Start with learning Git and GitHub for version control.
Understand CI/CD Pipelines: Tools like Jenkins, GitLab CI, or GitHub Actions are essential to automate building, testing, and deploying code.
Master Containerization: Tools like Docker help package applications into containers, ensuring consistency across environments.
Learn Infrastructure as Code (IaC): Tools like Terraform or AWS CloudFormation allow you to manage your infrastructure in a version-controlled, code-based way.
Explore Cloud Providers: Learn about cloud platforms like AWS, Azure, or Google Cloud to host and scale applications.
Set up Monitoring and Logging: Tools like Prometheus, Grafana, and ELK Stack help monitor and track your system's performance.
Understand Configuration Management: Tools like Ansible and Chef are used to automate infrastructure configuration.

Best Practices in DevOps

To ensure success in DevOps, consider following these best practices:

Automation First: Automate repetitive tasks, from testing to deployment.
Collaborative Culture: Foster open communication and collaboration between teams.
Continuous Integration and Continuous Delivery (CI/CD): Ensure your code is always tested and ready for deployment.
Monitoring and Feedback Loops: Always monitor your systems and applications to catch issues early.
Security by Design: Integrate security into the development process, not as an afterthought.

Who Can Learn DevOps?

DevOps isn't limited to developers or operations professionals—anyone interested in improving software delivery can learn DevOps. Whether you’re a software engineer, a systems administrator, a network engineer, or even someone just entering the tech industry, DevOps skills can be beneficial.

Prerequisites:

Basic knowledge of coding and scripting.
Familiarity with Linux or any other operating system.
Understanding of networking and cloud platforms is helpful but not mandatory.

Skills to be Used in DevOps

Here are the most important skills to focus on when learning DevOps:

Coding and Scripting: Knowledge of programming languages (Python, Bash, etc.) is essential for automating tasks.
Version Control: Expertise in Git and repository hosting services like GitHub or GitLab.
Continuous Integration/Continuous Deployment (CI/CD): Tools like Jenkins, CircleCI, and GitLab CI.
Containerization: Knowledge of Docker and Kubernetes for orchestrating containers.
Infrastructure as Code (IaC): Skills in tools like Terraform, Ansible, or CloudFormation.
Monitoring and Logging: Using tools like Prometheus, Grafana, or ELK for tracking system health.

DevSecOps and MLOps: Expanding the DevOps Ecosystem

DevSecOps

DevSecOps integrates security practices into the DevOps process. Rather than addressing security late in the development cycle, DevSecOps ensures that security is part of every phase—from design to deployment. This includes automating security checks like vulnerability scans, code analysis, and compliance checks.

MLOps

MLOps (Machine Learning Operations) focuses on deploying and maintaining machine learning models in production. Just as DevOps streamlines software development, MLOps does the same for machine learning by automating model deployment, monitoring, and retraining. It combines DevOps practices with data science to ensure efficient, scalable, and repeatable workflows.

Conclusion

DevOps is more than just a set of tools—it's a cultural shift that improves collaboration, automation, and efficiency in the software development lifecycle. Whether you're an aspiring engineer or a seasoned professional, learning DevOps can open doors to faster software delivery, better quality, and more secure applications. As technology evolves, incorporating practices like DevSecOps and MLOps will become increasingly important in creating reliable and scalable systems.

So, dive in, start learning, and embrace the future of software development with DevOps!

DevOpsifying a Go Web Application: An End-to-End Guide

Amash Ansari — Thu, 05 Sep 2024 18:55:47 +0000

Introduction

In this post, I will guide you through the process of DevOpsifying a Go-based web application. We will cover everything from containerizing the application with Docker to deploying it on a Kubernetes cluster (AWS EKS) using Helm, setting up continuous integration with GitHub Actions, and automating deployments with ArgoCD. By the end of this tutorial, you'll have a fully operational, CI/CD-enabled Go web application.

Prerequisites

Before starting this project, ensure you meet the following prerequisites:

AWS Account: You need an active AWS account to create and manage your EKS cluster for deploying the Go-based application.

DockerHub Account: You should have a DockerHub account to push your Docker images.

Basic DevOps Knowledge: Familiarity with DevOps concepts and practices is essential, including understanding CI/CD pipelines, containerization, orchestration, and cloud deployment.

Helm: Basic knowledge of Helm, the Kubernetes package manager, will be required to package and deploy your application.

By meeting these prerequisites, you'll be well-prepared to follow the steps in this guide and successfully DevOpsify your Go-based application!

Step 1: Getting the Source Code

To get started with the project, you'll need to clone the source code from the GitHub repository. Use the following command to clone the project:

git clone https://github.com/iam-veeramalla/go-web-app-devops.git

This repository contains all the necessary files and configurations to set up and deploy the Go-based application using the DevOps practices described in this guide. Once cloned, you can navigate through the steps below and follow them to containerize, deploy, and manage the application.

Step 2: Containerizing the Go Web Application

The first step is to containerize our Go application. We will use a multistage Dockerfile to build the Go application and create a lightweight production-ready image.

FROM golang:1.22.5 as build

WORKDIR /app

COPY go.mod .

RUN go mod download

COPY . .

RUN go build -o main .

FROM gcr.io/distroless/base

WORKDIR /app

COPY --from=build /app/main .

COPY --from=build /app/static ./static

EXPOSE 8080

CMD ["./main"]

Commands to Build and Push Docker Image:

docker login
docker build . -t go-web-app
docker push go-web-app:latest

In this Dockerfile, the first stage uses the Golang image to build the application. The second stage uses a distroless base image, which is much smaller and more secure, containing only the necessary files to run our Go application.

Step 3: Deploying on Kubernetes with AWS EKS

Next, we will deploy our containerized application to a Kubernetes cluster. Here’s how you can set up your cluster and deploy your app.

Create an EKS Cluster:

eksctl create cluster --name demo-cluster --region us-east-1

Deployment Configuration (deployment.yaml):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-web-app
  labels:
    app: go-web-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-web-app
  template:
    metadata:
      labels:
        app: go-web-app
    spec:
      containers:
      - name: go-web-app
        image: iamamash/go-web-app:latest
        ports:
        - containerPort: 8080

Service Configuration (service.yaml):

apiVersion: v1
kind: Service
metadata:
  name: go-web-app
  labels:
    app: go-web-app
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: go-web-app
  type: ClusterIP

Ingress Configuration (ingress.yaml):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: go-web-app
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: go-web-app.local
    http:
      paths: 
      - path: /
        pathType: Prefix
        backend:
          service:
            name: go-web-app
            port:
              number: 80

Apply the configurations using kubectl:

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl apply -f ingress.yaml

Setting Up Nginx Ingress Controller:

An Ingress controller in Kubernetes manages external access to services within the cluster, typically handling HTTP and HTTPS traffic. It provides centralized routing, allowing you to define rules for how traffic should reach your services. In this project, we use the Nginx Ingress controller to efficiently manage and route traffic to our Go-based application deployed in the Kubernetes cluster.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.1/deploy/static/provider/aws/deploy.yaml

Step 4: Packaging with Helm

To manage our Kubernetes resources more effectively, we package our application using Helm, a package manager for Kubernetes.

Create a Helm Chart:

helm create go-web-app-chart

After creating the chart, replace everything inside the templates directory with your deployment.yaml, service.yaml, and ingress.yaml files.

Update values.yaml: The values.yaml file will contain dynamic values, like the Docker image tag. This tag will be updated automatically based on the GitHub Actions run ID, ensuring each deployment is unique.

# Default values for go-web-app-chart.
replicaCount: 1

image:
  repository: iamamash/Go-Web-App
  pullPolicy: IfNotPresent
  tag: "10620920515" # Will be updated by CI/CD pipeline

ingress:
  enabled: false
  className: ""
  annotations: {}
  hosts:
    - host: chart-example.local
      paths:
        - path: /
          pathType: ImplementationSpecific

Helm Deployment:

kubectl delete -f k8s/.
helm install go-web-app helm/go-web-app-chart
kubectl get all

Step 5: Continuous Integration with GitHub Actions

To automate the build and deployment of our application, we set up a CI/CD pipeline using GitHub Actions.

GitHub Actions Workflow (.github/workflows/cicd.yaml):

name: CI/CD

on:
  push:
    branches:
      - main
    paths-ignore:
      - 'helm/**'
      - 'README.md'

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout repository
      uses: actions/checkout@v4

    - name: Set up Go 1.22
      uses: actions/setup-go@v2
      with:
        go-version: 1.22

    - name: Build
      run: go build -o go-web-app

    - name: Test
      run: go test ./...

  push:
    runs-on: ubuntu-latest
    needs: build
    steps:
    - name: Checkout repository
      uses: actions/checkout@v4

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v1

    - name: Login to DockerHub
      uses: docker/login-action@v3
      with:
        username: ${{ secrets.DOCKERHUB_USERNAME }}
        password: ${{ secrets.DOCKERHUB_TOKEN }}

    - name: Build and Push action
      uses: docker/build-push-action@v6
      with:
        context: .
        file: ./Dockerfile
        push: true
        tags: ${{ secrets.DOCKERHUB_USERNAME }}/go-web-app:${{github.run_id}}

  update-newtag-in-helm-chart:
    runs-on: ubuntu-latest
    needs: push
    steps:
    - name: Checkout repository
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.TOKEN }}

    - name: Update tag in Helm chart
      run: |
        sed -i 's/tag: .*/tag: "${{github.run_id}}"/' helm/go-web-app-chart/values.yaml

    - name: Commit and push changes
      run: |
        git config --global user.email "ansari2002ksp@gmail.com"
        git config --global user.name "Amash Ansari"
        git add helm/go-web-app-chart/values.yaml
        git commit -m "Updated tag in Helm chart"
        git push

To securely store sensitive information like DockerHub credentials and Personal Access Tokens (PAT) in GitHub, you can use GitHub Secrets. To create a secret, navigate to your repository on GitHub, go to Settings > Secrets and variables > Actions > New repository secret. Here, you can add secrets like DOCKERHUB_USERNAME, DOCKERHUB_TOKEN, and TOKEN. Once added, these secrets can be accessed in your GitHub Actions workflows using ${{ secrets.SECRET_NAME }} syntax, ensuring that your sensitive data is securely managed during the CI/CD process.

Step 6: Continuous Deployment with ArgoCD

Finally, we implement continuous deployment using ArgoCD to automatically deploy the application whenever changes are pushed.

Install ArgoCD:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'
kubectl get svc argocd-server -n argocd

Setup ArgoCD Project: To access the ArgoCD UI after setting it up, you first need to determine the external IP of the node where ArgoCD is running. You can obtain this by running the command:

kubectl get nodes -o wide

Next, get the port number at which the ArgoCD server is running using:

kubectl get svc argocd-server -n argocd

Once you have the external IP and port number, you can access the ArgoCD UI by navigating to http://<node-external-IP>:<port>. For example, if the external IP is 54.161.25.151 and the port number is 30498, the URL to access ArgoCD UI would be http://54.161.25.151:30498.

To log in to the ArgoCD UI for the first time, use the default username admin. The password can be retrieved from the ArgoCD secrets using:

kubectl edit secret argocd-initial-admin-secret -n argocd

Copy the encoded password from the data.password field and decode it using base64:

echo <encoded-password> | base64 --decode

For example, if the encoded password is kjasdfbSNLnlkaW==, decoding it with:

echo kjasdfbSNLnlkaW== | base64 --decode

will provide the actual password. Be sure to exclude any trailing % symbol from the decoded output when using the password to log in.

Now, after accessing the ArgoCD UI, since both ArgoCD and the application are in the same cluster, you can create a project. To do this, click on the "New App" button and fill in the required fields, such as:

App Name: Provide a name for your application.
Sync Policy: Choose between manual or automatic synchronization.
Self-Heal: Enable this option if you want ArgoCD to automatically fix any drift.
Source Path: Enter the GitHub repository URL where your application code resides.
Helm Chart Path: Specify the path to the Helm chart within your repository.
Destination: Set the Cluster URL and namespace where you want the application deployed.
Helm Values: Select the appropriate values.yaml file for your Helm chart.

After filling in these details, click on "Create" and wait for ArgoCD to create the project. ArgoCD will pick up the Helm chart and deploy the application to the Kubernetes cluster for you. You can verify the deployment using:

kubectl get all

That's all you need to do!

Conclusion

Congratulations! You have successfully DevOpsified your Go web application. This end-to-end guide covered containerizing your application with Docker, deploying it with Kubernetes and Helm, automating builds with GitHub Actions, and setting up continuous deployments with ArgoCD. You are now ready to manage your Go application with full CI/CD capabilities.

Feel free to leave your comments and feedback below! Happy DevOpsifying!

Reference

For a detailed video guide on deploying Go applications on AWS EKS, check out this video.