DEV Community: Rajit Paul

Enhance your Code Security with Amazon Inspector

Rajit Paul — Sun, 27 Jul 2025 15:47:03 +0000

As a latest addition to the vulnerability scanning capabilities of Amazon Inspector across multiple AWS services, it now supports scanning of your application source code, dependencies and Infrastructure as Code (IAC). It has a native integration with your SCMs - GitHub and GitLab and it helps you build a shift left security approach while taking proactive decisions securing your SDLC.

Let's Get Up and Running

Currently Amazon Inspector Code Security scan is available in 10 AWS Regions, you can get the full list here.

Considering you are in one of the listed regions and have activated Inspector, select Code Security on the left pane.

Once I click on ConnectTo, I get two options. As I am using GitHub as my SCM, I select it and proceed.

You can choose the default scan configuration or customise it according to your usecase. I will create a custom scan configuration for now.

In here you fix a scan frequency, I chose change based and periodic scanning which means whenever you create a pull or merge request or push new code a scan will be triggered. Alongside you can set weekly or monthly periodic scans, and in weekly scans you can choose the day you want the scan to run, this could be based off of your release cycles.

Next, you choose the scope of the scan analysis, in my case I need all the three options enabled, so I will keep things as it is and create the scan configuration.

Now I will provide a name to my configuration and connect to Github.

I will use the link mentioned in the pop up screen to authorize to GitHub.

Accept the authorization

Once authorized, I got a message on the top of Inspector Console asking me to visit the GitHub connections page

I will install a new GitHub App

Installed the app with a selected repository from my personal account.

We have a successful GitHub connection.

I pushed a commit to Master to trigger a scan.

Once the analysis is concluded (it might take some time), you can see the Scan status as Active against your Code Repositories.

In the Findings section you can see all the vulnerabilities in your code and you can get assisted remediation and fix with other details when you select the particular vulnerability. Also you can filter out the vulnerabilites based on SCM provider, severity, etc.

Testing Terraform IAC

Added one more repo with Terraform code to my GitHub application and it is listed in Code Security Console but pending inital scan, let's push a commit to the repository.

Code Security scanned the newly added repo and flagged issues in the code and also suggested remediation with code fixes which is super useful.

I created a seperate branch in my newly added repository and updated the Terraform code. Post which I created a PR to merge the changes with main, which triggered a CodeSecurity Scan on the GitHub console and once concluded it highlighted the code snippets that needed to be checked and stated the reason for flagging those with severity.

On-Demand Scan

I added a third repository to my GitHub Application, and it is listed in my Code Security console. This time instead of pushing some code or creating a PR, I will generate an On-Demand Scan for the repository.

I get a message saying that the On-Demand Scan generation is successful.

Within sometime I see findings generated :)

Lets Talk About Pricing

You are charged for each scan and each scan type 0.15 USD, against a single repository.

So a scan of a single repository with all three scan types enabled would cost 1*0.15*3 = 0.45 USD

If your repository does not contain IAC, you should create a new scanning configuration with IAC disabled to save costs.

Also there is an option in your scanning configuration to disable scanning when code is changed or disable periodic scanning if you want to save further costs and just rely on On-Demand scans, but this will not let you utilize the full potential of this tool.

More details related to pricing here.

Wrapping Up

I feel this feature is a great addition to the current capabilities of Amazon Inspector by helping find code vulnerabilities and misconfigurations early in the development lifecycle and I hope this blog will help you get started with Code Security 🤘

Elevating Security with Amazon GuardDuty Runtime Monitoring

Rajit Paul — Sat, 25 Jan 2025 07:12:46 +0000

With the majority of our applications now being cloud-native and containerized, ensuring security has become paramount. While static security measures, such as image scanning with Amazon Inspector, play a crucial role, monitoring container security during runtime is equally important. This is where ECS Runtime Monitoring with Amazon GuardDuty comes into play. GuardDuty Runtime Monitoring, now over a year in general availability, has proven its effectiveness in detecting runtime security threats across EC2 instances, ECS Clusters, and EKS Clusters. In this blog, we'll walk through enabling runtime monitoring for your ECS Cluster, generating GuardDuty findings, and setting up alerts for both runtime monitoring health and GuardDuty Findings to enhance your security posture.

Amazon GuardDuty: Advanced Threat Detection for AWS Security

Amazon GuardDuty is a fully managed threat detection service that continuously monitors your AWS environment for suspicious activity. By analyzing vast amounts of data from sources like AWS Cloudtrail, VPC Flow Logs, and DNS logs, GuardDuty detects threats such as unauthorized access, data exfiltration, or compromised instances engaging in malicious activity.

Leveraging AI, machine learning, and threat intelligence , GuardDuty identifies anomalies such as unusual login attempts, unexpected changes to resources, or attempts to disable security controls helping you respond before threats escalate. It provides automated analysis and actionable insights without the need for complex security infrastructure, making it an efficient and scalable solution for cloud security.

GuardDuty offers specialized protection across AWS Services including:

S3 Protection - Detects unauthorized access and data theft from S3 Buckets.
EKS Protection - Monitor Kubernetes workloads for suspicious activity.
Runtime Monitoring - Identifies real time threats in compute environments.
Malware Protection - Scans Amazon EC2 and S3 for malware threats.
RDS Protection - Guards against database related security risks.
Lambda Protection - Monitors serverless workloads for anomalies.

By automating threat detection and reducing manual security efforts, GuardDuty helps businesses safeguard their AWS infrastructure with minimal operational overhead.

Enabling the fully managed GuardDuty Agent

When we deploy the GuardDuty security agent, GuardDuty will create a VPC Endpoint for the security agent to deliver runtime security events to GuardDuty. Alongside it will also create a new security group that will control the traffic that's allowed to reach the resources using inbound rules of the security group and will adapt to vpc cidr range changes.

ECS Cluster

I started with an existing ECS Cluster with a single task running on AWS Fargate.

Within the task configuration, you'll notice two containers running:

Main Application Container
Sidecar Container launched by AWS to run the Amazon GuardDuty agent

GuardDuty actively monitoring the ECS Cluster

GuardDuty Runtime Monitoring Alerts

It is essential to configure alerts for when GuardDuty Runtime Monitoring enters an unhealthy state or when a Runtime Monitoring Finding is detected.

To achieve this, I have configured EventBridge rules with Amazon SNS as the target to trigger email notifications for both.

GuardDuty Runtime Monitoring Unhealthy State Alert

I manually scaled down the ECS service from 1 to 0, so that the GuardDuty agent is no longer able to communicate with Amazon GuardDuty and the Runtime Monitoring status is pushed to an unhealthy state, upon which the Eventbridge Rule is triggered and a SNS notification is generated.

Event Pattern for Eventbridge Rule:

{
  "source": ["aws.guardduty"],
  "detail-type": ["GuardDuty Runtime Protection Unhealthy"]
}

GuardDuty Runtime Monitoring Findings Alert

I generated sample findings in GuardDuty to test and validate the alerting mechanism.

Event Pattern for Eventbridge Rule:

{
  "source": ["aws.guardduty"],
  "detail": {
    "type": ["Backdoor:Runtime/C&CActivity.B", "PrivilegeEscalation:Runtime/DockerSocketAccessed"]
  }
}

You can find the full list of GuardDuty Runtime Monitoring Finding Types here.

Sample Findings Generated

Using an Encrypted SNS Topic

If you would like to encrypt your SNS Messages before saving them in it's data centers in order to comply with a certain compliance, there are a few things you need to ensure so that your GuardDuty alerts don't fail to deliver.

Firstly, you need to use a CMK(Customer Managed Key) instead of a default SNS Encryption key to encrypt your SNS Topic.
Secondly the Eventbridge rule should have the necessary permission to invoke your KMS key to decrypt the data.

While you are creating the Eventbridge rule note down the IAM role that is being created by default and you can later add the necessary permissions to it.

{
    "Effect": "Allow",
    "Action": [
      "kms:GenerateDataKey",
      "kms:Decrypt"
    ],
    "Resource": [
      "arn:aws:kms:ap-south-1:123456734:key/53c1b423-3a5e-1234-1111-eda13df344de"               ]
}

Replace your kms key arn!

Third, you need to add the necessary permission in your KMS policy to authorize Eventbridge.

If you see the messages not getting delivered, it means you failed to satisfy any of the above three pointers and for further troubleshooting you can also refer [this].(https://repost.aws/knowledge-center/sns-not-getting-eventbridge-notification).

Conclusion

I hope this blog serves as a great starting point for exploring this exciting feature. Below, I've compiled a few additional resources that will help you dive deeper and make the most of it.

Optimize Cost Savings using AWS EC2 Spot Instances as your EKS Worker Nodes

Rajit Paul — Sun, 20 Aug 2023 09:29:26 +0000

To optimize cost-savings while deploying dev/test workloads on EKS you can utilize Amazon EC2 Spot Instances and run them as your EKS Nodes.

Amazon EC2 Spot Instances let you take advantage of unused EC2 capacity in the AWS cloud. Spot Instances are available at up to a 90% discount compared to On-Demand prices. [Source: AWSDocs]

Pre-Requisites

An AWS Account
An IAM user with administrator access and a EC2 Role with administrator access

We are going to deploy an EKS Cluster using eksctl from an EC2 Instance which is going to be our launchpad, you can do the same from your local machine.

Launch an EC2 Instance and install necessary packages

We shall be launching an EC2 using the Amazon Linux 2023 AMI, with t3a.small instance type and keeping the rest of the settings default, if you wish you can change them based on your requirements. I've kept the SSH Access allowed for anywhere for the sake of this demo, highly recommend you to opt granular access for the same using MyIP.

Installing eksctl

For Unix:

# for ARM systems, set ARCH to: `arm64`, `armv6` or `armv7`
ARCH=amd64
PLATFORM=$(uname -s)_$ARCH

curl -sLO "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_$PLATFORM.tar.gz"

# (Optional) Verify checksum
curl -sL "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_checksums.txt" | grep $PLATFORM | sha256sum --check

tar -xzf eksctl_$PLATFORM.tar.gz -C /tmp && rm eksctl_$PLATFORM.tar.gz

sudo mv /tmp/eksctl /usr/local/bin

Source: eksctl docs

Installing kubectl
As we shall be launching the latest version of EKS (1.27) for amd64 based architecture, we will run the below commands

curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/1.27.1/2023-04-19/bin/linux/amd64/kubectl
chmod +x ./kubectl
mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$HOME/bin:$PATH
echo 'export PATH=$HOME/bin:$PATH' >> ~/.bashrc
kubectl version --short --client

Source: AWS Docs

Launching an EKS Cluster with spot instances using eksctl

ClusterConfig:

---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
    name: my-eks-cluster
    region: ap-south-1
    version: "1.27"

vpc:
  subnets:
    private:
      private-ap-south-1a: 
        id: "xxxxxxx"
      private-ap-south-1b: 
        id: "xxxxxxx"
      private-ap-south-1c: 
        id: "xxxxxxx"

managedNodeGroups:
    - name: spot-nodegroup
      ami: ami-016931097ac39b652
      amiFamily: AmazonLinux2
      overrideBootstrapCommand: |
        #!/bin/bash
        /etc/eks/bootstrap.sh my-eks-cluster --container-runtime containerd
      privateNetworking: true
      minSize: 1
      maxSize: 3
      desiredCapacity: 1
      instanceTypes: ["t3.medium","t3.small","t3a.small","t3a.medium"]
      spot: true
      subnets:
      - private-ap-south-1a
      - private-ap-south-1b
      - private-ap-south-1c
      labels: {node: spot}
      ssh:
        publicKeyName: yourkeypairname
...

Additionally we have to create an Admin Role for our EKS LaunchPad Server and attach it

To create the cluster, run eksctl create cluster -f cluster.yaml

When we create the cluster using eksctl, AWS launches two CloudFormation Stacks in the backend, one to create the control plane with additional infrastructure and the other to create the nodegroups.

It shall take from 20-25 mins to launch the cluster.

EKS Cluster Successfully Launched with Spot Instance NodeGroup

Clean-Up

Delete the cluster

eksctl delete cluster -f cluster.yaml

Terminate the EC2 Instance

Access AWS Secrets Manager from your container using AWS SDK

Rajit Paul — Sat, 19 Aug 2023 12:03:33 +0000

In case you need to store your credentials securely at a place and not in your application code, AWS Secrets Manager can become your ideal choice.

AWS Secrets Manager helps you manage, retrieve, and rotate database credentials, application credentials, OAuth tokens, API keys, and other secrets throughout their lifecycles.
[Source: AWS Docs]

Today, we are going to look into how to fetch a secret from AWS Secrets Manager inside your container using AWS SDK, we shall be using the Python SDK (boto3). I shall be going ahead with a dummy secret for this demo but you can use the same process to fetch DB Passwords, Application Credentials or other critical tokens that you should not hardcode onto your application source code.

Pre-Requisites

An AWS Account
An user with full access to AWS Secrets Manager and EC2

Creating a secret in the Secret Manager

We shall be going ahead with other type of secrets but in your case you can go ahead and store secrets if you are using AWS Native Databases services as well.
We have chosen aws/secretsmanager as the Encryption Key, you can have a Customer Managed KMS Key to encrypt yoru secret based on your requirement.

In the next window, you shall be asked to provide a secret name in our case we have provided test/mysecret, you can leave the rest of the options as default.

Click on next, and if you wish to enable automatic rotation you can do so in this window, this would also require a lambda function that will rotate the secret.

Click next and in the Review section you shall be getting a code snippet for multiple languages, according to your needs you can choose one, in this case I shall be going ahead with Python3 and we shall use that later.

Launching an EC2 Instance and installing Docker

Create an instance providing the name and selecting the instance type.

Choose an instance type and your keypair, if you don't have a keypair you can create one using the create keypair option.

You can keep the network settings as default, for this demo I'm keeping the SSH Access open from anywhere, it's recommended to keep restricted access from the same.

Once the instance is launched you can ssh into the instance and install docker

Start the docker service

Creating an IAM Role for EC2 to access Secrets Manager

Select EC2 as the trusted entity type

Choosing the SecretsManager R/W Permission, in your case you can choose a granular permission

Provide a role name and create the role

Attach the role to your EC2 Instance

Launch an Ubuntu Container and Access Secrets Manager

sudo docker run -it ubuntu /bin/bash

Install Python3 and boto3 in the container

apt update && apt install python3 -y

apt install python3-pip -y

pip3 install boto3

We shall also install vim in the container using - apt install vim -y

Access Secrets Manager inside the container using a Python Script

We shall use the code snippet we got while creating the secret and add a command to print the secret, and subsequently a call statement to call the get_secret method.

https://raw.githubusercontent.com/RajitPaul11/AWS-Security/main/access-secrets-manager-using-boto3.py

Output

CleanUp

Terminate the EC2 Instance and schedule deletion for the secret, the minimum duration is 7 days.

Monitor and Visualize Nginx Ingress Controller Metrics on Amazon EKS with Prometheus & Grafana

Rajit Paul — Thu, 25 May 2023 12:17:59 +0000

In today’s digital landscape, it is very necessary to figure out trends in our data and act accordingly to ensure high availability of our application. Monitoring helps us to stay on top of the game and get insights on our product environment. An Ingress Controller acts as a bridge between Kubernetes Service and the external world and can be considered as a specialized load balancer for Kubernetes.

Pre-Requisites

A Kubernetes Cluster (Create an EKS Cluster from scratch)
Helm V3 (Helm Install Docs)

Install Kube-Prometheus-Stack using Helm

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 
helm repo update 
helm install --create-namespace --namespace monitoring \
my-k8s-prom-stack prometheus-community/kube-prometheus-stack

Once you have the stack installed you should have Prometheus, Grafana and AlertManager installed onto your cluster.

Install Nginx Ingress Controller

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx 
helm repo update 
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx --create-namespace \
--set controller.metrics.enabled=true \
--set controller.metrics.serviceMonitor.enabled=true \
--set controller.metrics.serviceMonitor.additionalLabels.release="my-k8s-prom-stack" \
--version=4.5.2

Along with installing the ingress controller we are setting the controller metrics as true which will populate an additional metrics service and creating a service monitor for that service which will feed data onto Prometheus, the additional Label would be as per the selector you have set on the Prometheus Operator, follow the later part for more details on this.

You can check the Service Monitor selector on your Prometheus Operator using:

kubectl get prometheus --namespace monitoring prometheus-svc-name –oyaml (lookout for the section below on the manifest)

Once Nginx controller is installed you can verify the services that are spawned using:

kubectl get svc --namespace ingress-nginx

Monitor and Visualize Nginx Controller Metrics

To access the Prometheus and Grafana you can set ingress objects
Sample Ingress Manifest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:    
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 5m  
  name: prometheus-ingress
  namespace: monitoring
spec:
  rules:
  - host: prom.mydomain.com
    http:
      paths:
      - backend:
          service:
            name: my-k8s-prom-stack-kube-pro-prometheus
            port:
              number: 9090
        path: /
        pathType: ImplementationSpecific

Generate some load on the Ingress Controller by hitting the Ingress Endpoints.

Access Prometheus and Grafana to query and visualize the controller metrics:

Prom Query to check the number of 200 requests on ingress:
nginx_ingress_controller_requests{status=~'2..'}

Import Nginx Controller Dashboard on Grafana and visualize the metrics (Dashboard ID: 9614)

Thank you for reading this, you can follow me for more such content, or reach out to me on Linkedin.

Deploy your application on Kubernetes (Amazon EKS) using AWS Serverless (Codebuild)

Rajit Paul — Thu, 21 Jul 2022 14:52:32 +0000

Hi folks,
Recently I came across an use-case of deploying a microservice on EKS using CodeBuild with GitHub as source. Although I've used Jenkins numerous times to do the same thing but I haven't used AWS Serverless to deploy on EKS.

This blog is for you if you want to deploy your microservice to Kubernetes, or want to learn how to setup AWS CodePipeline with CodeBuild, or like to integrate CodeBuild with EKS or you are generally curious about Kubernetes and Serverless :)

This is how I started the journey:

I enquired about the source code and got to know it was on GitHub. For this blogpost I am going to create my own GitHub Repo with a basic deployment manifest. You can create your own repo and have the full stack of manifests starting from Ingress, to the service, deployment etc required for your microservice.

Next, I setup a CodePipeline. I'll show you how to do that:

Navigate to Developer Tools in the console, and select Code Pipeline. Provide your pipeline a name, select the default service role so CodePipeline can create a role on your behalf, let the advanced setting be as it is unless you have a custom location for your artifact and want to use a custom KMS Key.

Moving on you need to specify where is your SourceCode that you want to build or deploy.

Add a Source Provider: In our case that would be GitHub (Version2). Connection: Select if you have an existing connection with your GitHub Account or Create a connection for GitHub, it's fairly simple. Once you authenticate CodePipeline to connect to your GitHub Account you shall receive a CodeStar connection URL, use that. Once you fill in all the details it should look something like this.

My GitHub Repo

Next, we shall add a build stage.

As a build provider we will select AWS CodeBuild. Feel free to choose the region of your choice. If you have an existing project you can select the same or else create a new project. I am going to setup a project from scratch.
To setup a CodeBuild Project you need to provide the Project Name & Description, you can also restrict concurrent builds and provide additional tags if you want.
Design the CodeBuild Environment, we shall go with the latest image of Amazon Linux 2 and ask CodeBuild to create a new service role on our behalf.
Additional Environment Configuration
Select your VPC, choose a private subnet and a SG with outbound allow and then validate your VPC setting. This is where your codebuild server will be provisioned ~ No, Serverless does not mean there are no servers, it's just that, you don't have to manage them ;)
Provide appropriate compute resource to the server as per your code requirements and we are good to go. If you require you can add environment variables and filesystems for your server.
You can leave the buildspec section empty if your buildspec file is buildspec.yml as codebuild will look for that file in your repo, if you have named your file otherwise you can mention that in the buildspec name section. Also if you have some additional requirements while building your code you can mention those in the additional build commands.
Will ignore Batch configuration as we do not require that for this blogpost.
It's best practise to export your build logs to Cloudwatch so that it's easier for you to troubleshoot. Additionally you can also export your CodeBuild logs to S3 for later analysis.

Once you click on continue to CodePipeline, the CodeBuild Project will be created and you can complete your CodePipeline setup.

In environment variables you can refer to environment values generated from CodePipeline or can add new env variables. On Build Type, we shall be executing a single build on execution.

Skip Deploy stage as codebuild will be taking care of the deployment and create your CodePipeline :)

EKS Cluster: I shall consider you have a running EKS cluster where we shall be doing the deployment, if not, you can deploy a new EKS cluster. In case you need help, refer this previous blog of mine - Setup your EKS Cluster from scratch

To allow CodeBuild to deploy on the EKS cluster we need to modify EKS RBAC by adding the CodeBuild Service Role with the required permission on the aws-auth configmap that is used to manage EKS RBAC.
To know more about EKS RBAC

Locate your CodeBuild Service Role:

Open your buildproject, that you shall be using. In Build details tab, scroll down to Environment where you can see the Service Role hyperlink.

Update the aws-auth configmap:

To edit the configmap run - kubectl edit cm aws-auth -n kube-system

Under mapRoles we shall add a new entry:

- groups:
    - system:masters
  rolearn: arn:aws:iam::xxxaccidxxx:role/codebuild- microservice-deploy-to-eks-service-role 
  username: CodeBuild Role to Access EKS

Note If you directly copy paste the CodeBuild Role ARN from the console to the configmap you will get a "error: You must be logged in to the server (Unauthorized)", make sure your remove the /servicerole path from the ARN.

Additionally to the CodeBuild Service Role attach a policy with eks:DescribeCluster action allowed. This will allow codebuild to download the kubeconfig file onto it's server.

Deploy Your Application, Run the Pipeline:

Once you have done all as I mentioned, you would have your application running on EKS with the help of CodeBuild :)

Clean UP

Delete your CodeBuild Project
Delete your Pipeline

If you want to know more about Kubernetes, DevOps, Serverless follow me, also I would love to have a chat with you on LinkedIn :)

ECS Networking - (awsvpc, bridge, host, none)

Rajit Paul — Mon, 09 May 2022 14:47:06 +0000

Hi folks, Elastic Container Service is one of the container offerings from AWS. ECS helps us to run any number of docker containers across a managed cluster of EC2 instances. It helps to isolate our workloads and helps achieve faster time to market with efficient scaling in place. It is secure and you can easily migrate your on prem container workload to ECS and back.

Let's deep dive and look into the different network types on ECS and see how they are different from one another.

We have Four Network modes in ECS:

awsvpc: It allocates a seperate Elastic Network Interface (ENI) to the task and also allocates a primary IPV4 address to it. The task networking behaves same as an EC2 instance networking.

In this you can see a warning which says the containers in the task will share an ENI and port mappings can only specify container ports.

We cannot set host port mappings as the network mode is awsvpc.

Once you create the service we can check in the task, an ENI is assigned to the task and all the containers inside it.

If we SSH into the instance and curl the private IP associated to the task ENI, we can access the website running on the container.

In this network mode we cannot access the website using the Task Host (EC2) Public or Private IP.

bridge: In Bridge Network mode, the task makes use of the built-in Docker VNet (Virtual Network) which also allows the task to communicate with other tasks.

Once we select the bridged network mode for the task we can see an associated host port mapping available with the container port.

If we check task networking the container does not have any additional network as it uses only the Docker Virtual Network.

We shall access the website running on the container using the DockerHost IP (Amazon EC2).

host: Host network mode facilitates the task to bypass the Docker built-in VNet (Virtual Network) and maps the container port directly to the task host (Amazon EC2) ENI. As a result, we cannot run multiple instances of the same task when Port Mappings are used and the network mode is host.

The container shall be using in this case the instance network stack.

We can access the website running on the container using the Docker Host Public IP (EC2 Instance Public IP).

none: Blackhole, the task does not have any external network connectivity.

You shall see a message stating that the container will not have any external connectivity in the network section of the task.

I hope this has helped you get an idea of ECS networking. Follow me for more blogs on AWS & DevOps.
Feel free to connect with me on LinkedIn!

Dockerize an API based Flask app and deploy on Amazon ECS

Rajit Paul — Sun, 17 Apr 2022 19:46:52 +0000

Hi Folks!
This is the first blog of the series Dockerize Your Application.

In the age of microservices we want our application code and requirements to be packed in an image and use that in a suitable container orchestration tool for better scalability and availability.

Pre-Requisites:

An EC2 Server with git and docker installed
Amazon ECR Repository
AWS CLI configured on your server with sufficient permissions (to push image to ecr) or you can attach a role to your server with sufficient permissions.
Region: Mumbai(ap-south-1)

About the API

It's a todo app with two API's one to get the list of to-do tasks and another to post your tasks.
GET-POST api path - /todo/api/v1.0/tasks

Source: [Flask_API_App]

(https://github.com/RajitPaul11/AWS_workshop_2022_data/tree/collate/python_flask_code_in_aws_linux_restful_GET_POST)

Dockerfile

FROM alpine:latest

RUN apk add py3-pip
RUN pip3 install flask

WORKDIR /home

COPY app.py .

EXPOSE 80

ENTRYPOINT ["/usr/bin/flask","run"]

CMD ["--host=0.0.0.0", "--port=80"]

Login to your EC2 Server and clone the repo

git clone -b collate https://github.com/RajitPaul11/AWS_workshop_2022_data.git

Change directory and build docker image

cd AWS_workshop_2022_data/python_flask_code_in_aws_linux_restful_GET_POST
docker build -t flask_api_app:v1 .

Tag your docker image

docker tag flask_api_app:v1 youraccountID.dkr.ecr.ap-south-1.amazonaws.com/flask_api_app:v1

Login to your ECR Repo

aws ecr get-login-password --region ap-south-1 | docker login --username AWS --password-stdin youraccountID.dkr.ecr.ap-south-1.amazonaws.com

Push the docker image

docker push youraccountID.dkr.ecr.ap-south-1.amazonaws.com/flask_api_app:v1

Deploy to ECS

Select Services and then select ECS

Create an ECS Cluster

We shall be creating an ECS Cluster with EC2

Cluster Config

Choose a suitable name for your cluster, and select a provisioning model, in this case we shall go for Spot.
Choose diversified spot instance allocation strategy so the instances are spread across az's.
Select two instance types on a or basis.

Specify the Storage spec and select an existing key pair, so that you can ssh later to the EC2 instance and do some modification or troubleshoot from the terminal.
Create a new VPC or you can select an existing VPC.

Cluster created!

Create a Task Definition

Select the launch type as EC2

Provide a suitable task def name, and select the Task Role and network mode (we shall be looking into the different network modes in an upcoming blog, for now let's go ahead with bridge), select a task execution role.

Allocate sufficient task memory and cpu based on your application requirements.

Add a container

Provide a container name and the ecr repo uri along with the version, you can set hard limit for the container in case you have set the task cpu and memory req this is not required, if you want dynamic port mapping, keep the host port as 0, in this case we have set it to 80 same as the container port

As per requirement you can explore advanced details and set container healthcheck, container timeouts, storage, logging and more.

Create a Service

Select the launch type as EC2, select your task definition and it's version you can see the latest suffix to denote the latest version, select your cluster and provide a suitable service name, select a service type (in this case we shall go with replica), provide the number of tasks you want to run (keep in mind the instance type you chose and the resource allocated to each task while you designate the number of tasks)

Select the deployment strategy (we shall go into depth on this in an upcoming blog, in this case we choose Rolling Update), select a Task Placement strategy (AZ Balanced spread will help to spread tasks across instances in different AZ's for high availability)

Configure a Load Balancer

Select your load balancer type(in this case we choose application load balancer), Create a new service IAM role, and select your existing Load Balancer.

In your target group you can register the existing ECS instance, and set the health check path as /todo/api/v1.0/tasks

If you want to scale your tasks you can enable autoscaling, in this case we do not want to scale our tasks so we won't enable auto scaling.

Select your listener, target group name for the Load Balancer and rest shall be populated

Service Created and Task Running!

Allow Port 80 in EC2 and ALB

Test API using ALB URL

GET Request

PUT Request

If you have any queries you can connect with me on LinkedIn

Creating an Amazon EKS Cluster from scratch using eksctl

Rajit Paul — Wed, 30 Mar 2022 14:47:45 +0000

Hi folks, to create an EKS cluster, you require a launch pad, for today we shall be using an Amazon Linux 2 EC2 server as our eks launchpad. There are few pre-requisites we require to take care of -

kubectl: Kubernetes Client to communicate with the Kubernetes API Server.

Installing kubectl: [Source: Installing kubectl - AWS Docs]

curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.21.2/2021-07-05/bin/linux/amd64/kubectl
chmod +x ./kubectl
mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$PATH:$HOME/bin
echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc
kubectl version --client

eksctl: The official Amazon EKS CLI, used to create and manage multiple EKS Clusters.

Installing eksctl: [Source: Installing eksctl - eksctl docs]

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
export PATH=$PATH:/usr/local/bin
echo 'export PATH=$PATH:/usr/local/bin' >> ~/.bashrc
eksctl version

Create an IAM Role for EC2 with following IAM policies:
Source : eksctl doc

Once the Role is created you can attach the role onto your EKS Launch Pad Server.

Once the pre-requisites has been taken care of we can go ahead with cluster creation.

Create a file named cluster.yaml with the following configuration: Source: My GitHub - cluster.yaml
Run eks create cluster with dry run
eksctl create cluster -f cluster.yaml --dry-run

[ This shall help you identify any errors on the config files or related to your permission, make sure you don't have additional aws user configured with less privileges than the privileges allowed in the EC2 attached role. ]

Launch your cluster with eksctl create cluster -f cluster.yaml

You need to wait for a few minutes and you shall see on the screen the CFN Stack is being deployed

The CFN stack creates the EKS Control Plane, SG's, Policies and Service Roles. It also creates a single nodegroup or more as mentioned in the cluster config.

If you encounter any issues check the Cloudformation Console or try:
eksctl utils describe-stacks --region=Your-Region --cluster=Your-Cluster-Name

The EKS cluster has been successfully created 🎉

You can access the EKS cluster from your launch pad using kubectl!

Clean UP

To delete the EKS Cluster run:
eksctl delete cluster your-cluster-name

I hope you enjoyed the blog, if you face any issues please reach out to me on LinkedIn and we can discuss the same, thanks!

Wrap Up

You can follow me to get updated on new AWS related blogs in the coming weeks, also I am an earth buddy, don't know what that is, check this out: Save our Soil

Adios!

Restric Access to Cloudfront Distribution using Lambda@Edge

Rajit Paul — Sat, 26 Mar 2022 06:36:51 +0000

Hi folks!
Recently I came across a usecase where I had to restrict access to a website in the UAT environment, so that the iterations and changes to the UAT env are not available for public view. The website was served with the CloudFront CDN, so one of the ways we could restrict access was enforcing authentication on the Cloudfront Distribution using Lambda@Edge.

How did I achive this?

Create a Lambda Function

Go to us-east-1 region as that is the only region we can deploy Lambda@Edge functions.
Go to the AWS Console and choose Lambda from services.
Create function and author from scratch.
Provide function name, and select Node JS 14.x as runtime.
In Change Default Execution Role, create a new role with basic Lambda permissions.

Boom! Your function is created.

Use the below code and replace the user and password with your required username and password. code

Source: https://gist.github.com/njofce/3382b0fe51c59ae9038046cd5087e42a

Deploy the code.
Go to general configuration in Configuration and change the function timeout to 5sec, that's the max allowed timeout for CDN triggered Lambda Function.
Go to Permissions, and open the IAM role, and update the Trust Relationships with the below json snippet, which allows lambda and lambda@edge to assume the role.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "lambda.amazonaws.com", "edgelambda.amazonaws.com" ] }, "Action": "sts:AssumeRole" } ] }

Go to Actions, and under capabilities, select Deploy to Lambda@Edge.
Configure a new Cloudfront Trigger, select your distribution under Cloudfront event and select a Viewer request, then check include body, and confirm deploy to Lambda@Edge.

It shall take few minutes to enforce the authentication, once done we can see a sign in option as we try to access our UAT env website, provide the username and password you used in the code and access your website. (to not disclose our client, I have used a dummy CloudFront URL with my content)

If you face any challenges please connect and discuss with me on Linkedin

Also I'm currently part of the Save Soil Movement initiated by the Isha Foundation, I know you are aware of the conditions of the soil and as a generation we have to turn this around, Become an Earth Buddy
Stay Joyful! :)

Bitbucket Branch Based Generic Webhook Trigger on Jenkins

Rajit Paul — Wed, 02 Mar 2022 13:47:56 +0000

Namaskaram!
I faced a recent challenge when I used BitBucket Webhook to trigger a Jenkins build, the challenge was webhooks on bitbucket are based on the entire repository, which means a push to any of the BitBucket Branch will trigger a Jenkins build.
What was I actually looking for was a branch based webhook to trigger Jenkins Build.

What you'll need

Jenkins
BitBucket

Configuring BitBucket:

Go to the Repository Settings of BitBucket and then to Webhooks.
Add new webhook (Title, URL- http://JenkinsUsername:JenkinsPassword@Jenkins PublicIPorPublicDomain:8080/generic-webhook-trigger/invoke, Status is Active, Triggers on Repository Push).
Save it.
Once saved go to view requests, and enable request history collection.

Update code in the specific branch of BitBucket:

In this part you need to make some changes in the branch of BitBucket that you want to trigger the webhook from.

Once the code is updated it will trigger a webhook request.
Go to repository settings then to webhook, and then to view requests you should see something like this. (The status code may be different 404 is ok, but 301 means you are facing authentication issue)
Click on view details, and scroll down to the bottom, you can see the Request - headers and body, we need to expand the body.
Once the body is expanded, copy the entire body from top to bottom.
Go to https://jsonpath.curiousconcept.com/ and paste the JSON data you copied in your clipboard.
Run this Json Path Expression - push.changes[0].new.links.commits.href
In the result you should get the commit link with the repo and it's branch we are pushing to, keep this link saved. Our job here is done, now we move to Jenkins.

Configuring Jenkins:

First we shall install the Generic WebHook Trigger Plugin for Jenkins. Go to Manage Jenkins - Plugin Manager - and Install this plugin
Go to the Configuration of the Jenkins Job you wish to trigger using BitBucket.
In Build Triggers you should see Generic Webhook Trigger
Click on add Post content Parameters - Add a variable named branch, in the expression put the JSON Path Expression we used earlier ( push.changes[0].new.links.commits.href ) and select JSONPath.
Scroll down and in Optional Filter, provide the RegEx

[.+?(?=repository/commits/branch)] - replace the repository and branch with your repo and branch name.

You can refer the JSON Path Result that you saved earlier
https://api.bitbucket.org/2.0/repositories/project/repository/commits/branch, from this you shall get the repository and branch.

Considering you have your build setup, you can save the job.

Make some changes in the code in the specific branch and see your Jenkins Job being triggered!

If you wish to connect or face some issues while performing this demo, you can reach out @ https://www.linkedin.com/in/rajitpaul/
See ya!

Setup KinD (Kubernetes In Docker) on Amazon EC2

Rajit Paul — Mon, 21 Feb 2022 08:06:26 +0000

Install Docker on Linux -

As I'm using an Amazon Linux I can install docker with a
yum install docker -y

Start docker service - systemctl start docker

For Docker Installations - https://docs.docker.com/engine/install/

Install Go -

Go to https://go.dev/dl/
Copy the link of go tarfile for linux platform
Install using wget

Extract it using tar zxf go1.17.7.linux-amd64.tar.gz -C /usr/local

Add /usr/local/go/bin to the PATH variable.

Install KinD

go install sigs.k8s.io/kind@v0.11.1
You can replace v0.11.1 with the latest stable kind version

Move the KinD Binary to /usr/local/bin -

You can find the kind binary inside the directory go/bin
Move it to /usr/local/bin - mv go/bin/kind /usr/local/bin
Make sure you have a path setup for /usr/local/bin

Install Latest Version of Kubectl:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
mv kubectl /usr/local/bin

Create a cluster with kind

kind create cluster --name labcluster

kubectl get nodes -o wide

docker ps

You are now running KinD successfully in Linux
Once done you can delete the cluster using
kind delete cluster

Multi-Node Cluster -

Create a config file kind-example-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:

role: control-plane
role: worker
role: worker

kind create cluster --config kind-cluster-config.yaml

You now have a running KinD Cluster with one master and two worker nodes.
kubectl get nodes -o wide