DEV Community: Rasanpreet Singh

Automate CloudWatch Logs Export to S3 using Lambda and Event Bridge

Rasanpreet Singh — Mon, 19 Jun 2023 06:34:36 +0000

Overview

You can consolidate the logs from every system, application, and AWS service you use into a single, highly scalable service by using AWS CloudWatch Logs. For each log group, you may modify the retention policy, maintain indefinite retention, or select a retention time between one day and ten years.

If your organization has to report on CloudWatch data for a period longer than the designated retention duration, exporting CloudWatch data to an S3 bucket is a critical step in many hybrid environment scenarios. As S3 offers extremely long-lasting storage, it may also be connected with other monitoring or logging systems (like Microsoft Sentinel, for example). Additionally, keeping logs in CloudWatch with an extended retention period might be costly.

There are two ways to push CloudWatch Logs to S3:

Manual process

Using the AWS Management Console, AWS Command Line Interface (CLI), or an SDK, we may export log data to S3 directly without using any other AWS service. This technique is straightforward and perfect when we just need to export logs once. However, using this strategy, we are unable to automatically move CloudWatch logs to S3 after a specific interval. To learn more, kindly click the following link: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasksConsole.html

Automated process

In this article, we'll go over a step-by-step procedure for utilizing a Lambda function and event bridge to automate the process of exporting CloudWatch log group to an S3 bucket. Here, we've utilized a lambda function with an event bridge trigger on it to automate the export of CloudWatch logs to an S3 bucket as per schedule on the trigger when run every time, which automates the process.

Prerequisites

· We have one CloudWatch log group where logs are ingested through any AWS service or application.
· One Amazon S3 bucket with the same region as the cloud watch log group.
· One IAM lambda role with rights for S3, CloudWatch, and lambda execution.

The architecture diagram illustrates the complete deployment architecture, which includes a lambda function, an event bridge rule, CloudWatch logs and an S3 bucket.

Steps walkthrough

Create Lambda Function

Go to the AWS Lambda dashboard > Functions> Create function > Choose Author from scratch. Give function a name. Choose runtime as Python 3.7

Under permissions, choose an existing execution role with CloudWatch and S3 permissions and create a lambda function.
Please refer to the below snippet for lambda role permissions.

After creating the lambda function, go to Code window and put the following code there. After updating the following variables as per your environment,
GROUP_NAME = The name of the CloudWatch log group for which you want to export data.
DESTINATION_BUCKET = The name of the S3 bucket for the exported log data
PREFIX = the prefix used to begin each export object's key



import boto3
import os
import datetime

GROUP_NAME = "CloudWatch log group"
DESTINATION_BUCKET = "S3-bucket"
PREFIX = "S3-Prifix"
NDAYS = 1
nDays = int(NDAYS)

currentTime = datetime.datetime.now()
StartDate = currentTime - datetime.timedelta(days=nDays)
EndDate = currentTime - datetime.timedelta(days=nDays - 1)

fromDate = int(StartDate.timestamp() * 1000)
toDate = int(EndDate.timestamp() * 1000)

BUCKET_PREFIX = os.path.join(PREFIX, StartDate.strftime('%Y{0}%m{0}%d').format(os.path.sep))

def lambda_handler(event, context):
    client = boto3.client('logs')
    response = client.create_export_task(
         logGroupName=GROUP_NAME,
         fromTime=fromDate,
         to=toDate,
         destination=DESTINATION_BUCKET,
         destinationPrefix=BUCKET_PREFIX
        )
    print(response)

The lambda code screen will look as below

Create CloudWatch Event bridge rule.

Go to CloudWatch dashboard > Events > Create rule. Give the rule name. Choose rule type as schedule and select next.

In the schedule pattern option, set the rate expression to run every 5 minutes and go next.

Next, for target type, select AWS Service. Select the lambda function as the target and mention the lambda name that we created in the above steps. Verify the configurations and create a rule.

In order to allow CloudWatch to store items in the target S3 bucket, update the bucket policy. Add the following bucket policy with the correct S3 bucket ARN and AWS region details.



{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "uploadcloudwatchlogs",
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.us-east-1.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::s3-bucket"
        },
        {
            "Sid": "uploadcloudwatchlogs",
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.us-east-1.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3::: s3-bucket/*"
        }
    ]
}

Now the lambda function will execute every 5 minutes and push the CloudWatch logs to S3 buckets. After successful lambda execution, you can view the CloudWatch logs exported to the S3 bucket.

Deploying Kubernetes cluster with YAML on AWS EKS

Rasanpreet Singh — Sun, 14 May 2023 03:35:31 +0000

Overview
A managed service called Amazon EKS makes it easier to set up Kubernetes on AWS. In order to manage containerized workflows, Kubernetes gives businesses a wide range of advantages, therefore, all startup and enterprise IT organizations are trying to adopt it.
The official CLI tool for Amazon EKS is eksctl. It facilitates the creation of nodegroups and clusters. There are two ways to start an EKS cluster using eksctl.

• eksctl CLI and parameters by a single command. It is straightforward. Refer to this blog for more details. https://medium.com/aws-tip/build-aws-kubernetes-eks-cluster-with-eksctl-9040411badcb
• eksctl CLI and YAML configuration.

However, in this post, we favour the YAML configuration as it allows for declarative cluster configuration that is reusable. YAML files, which specify the requirements for the application and enable reproducible setup and deployment, are the source of Kubernetes' magic. Your Kubernetes cluster can be declaratively configured using this method. The fundamental benefit of YAML over other comparable formats is that it is human readable. In addition to building EKS clusters, eksctl also includes best practices for tagging, annotation, addons, policies, and other things.

Prerequisites
You must prepare and install the following requirements on your local system before you begin the EKS installation:
• AWS CLI
• Kubectl
• Eksctl
For installation steps details please refer the link shared in overview section

Create EKS Cluster
Create the YAML Recipe for the EKS Cluster

You must produce a YAML file with the EKS cluster's configuration information to utilize eksctl. You must set the following parameters in the yaml file:

name: mention name for the cluster
region: Name of the Amazon Region where the Cluster should be created.
vpc: The VPC where the cluster should be configured. A new VPC will be generated automatically if you don't set this.

Create a file named cluster.yaml
vim cluster.yaml

Copy the following contents to the file. You need to replace the region, cluster name, nodegroup name, SSH keypair name etc. as per your requirement.

`apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: eks-cluster
  region: us-east-1


managedNodeGroups:
  - name: eks-cluster-ng
    instanceType: t3.small
    minSize: 1
    maxSize: 3
    desiredCapacity: 2
    volumeSize: 20
    volumeType: gp3
    ssh:
      allow: true
      publicKeyName: kube-ssh
    tags:
      Env: Dev
      k8s.io/cluster-autoscaler/enabled: 'true'
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
      withAddonPolicies:
        autoScaler: true`

Run eks create cluster with dry run option. This will help to identify any errors on the config files or related to any permissions.

eksctl create cluster -f cluster.yaml --dry-run

Launch your cluster with following command.

eksctl create cluster -f cluster.yaml

Once the creation process has finished you will receive a message that says EKS Cluster "eks-cluster" in "us-east-1" region is ready:

The CloudFormation templates are deployed at the backend when you run the cluster creation command mentioned above. The clusters should ideally be deployed using CloudFormation templates. You can see that a CloudFormation Stack has been set up for provisioning control plane and node groups in the image below in the CloudFormation dashboard. we need to wait for 10-15 minutes.

*VPC information *
When using the above YAML, eksctl will automatically configure a new VPC in your AWS account with all the configurations and setup necessary for a EKS VPC.
If you needed to use an existing VPC, you can use a config file like this with updating the respective subnet id details:

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: eks-cluster
  region: us-east-1
  version: "1.24"

vpc:
  subnets:
    public:
      pub-ap-south-1a: 
        id: "xxxxxxxx"
      pub-ap-south-1b: 
        id: "xxxxxxxx"
      pub-ap-south-1c: 
        id: "xxxxxxxx"
managedNodeGroups:
  - name: eks-cluster-ng
    instanceType: t3.small
    minSize: 1
    maxSize: 3
    desiredCapacity: 2
    volumeSize: 20
    volumeType: gp3
    subnets:
      - pub-ap-south-1a
      - pub-ap-south-1b
      - pub-ap-south-1c
    ssh:
      allow: true
      publicKeyName: kube-ssh
    tags:
      Env: Dev
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
      withAddonPolicies:
        autoScaler: true

Connect to EKS cluster.

Once the cluster is provisioned, you can use the following AWS CLI command to get or update the kubeconfig file.

aws eks update-kubeconfig --region us-east-1 --name <cluster_name>

Verify EKS Cluster by following commands.

# Get List of clusters
              kubectl cluster-info
              eksctl get clusters --region us-east-1

Congratulations, your Amazon EKS Cluster is now operational and ready for usage!! You can verify the cluster status on the EKS dashboard.

Cleaning Up
We can delete the whole cluster (about 15 minutes) with this command:

eksctl delete cluster --name <clusterName> --region us-east-1

Network Security with Amazon Network Firewall

Rasanpreet Singh — Thu, 23 Mar 2023 12:04:13 +0000

Overview
At AWS, cloud security is "job zero" task. Security is given high attention while developing AWS services and features. Network-level traffic filtration is one level of defense that can be used to protect your network and computers from security events. For all Amazon Virtual Private Cloud Infrastructure, Network Firewall is a highly available, managed solution that makes it simple to establish essential network security. To safeguard your virtual networks on AWS, it helps you to quickly create and manage stateful inspection, intrusion prevention and detection, and web filtering. With no additional customer investment in security infrastructure, Amazon Network Firewall automatically scales with your traffic to ensure high availability.
AWS Network Firewall consists of three essential parts.

Rule Groups: It stores criteria for inspecting traffic and processing packets and traffic flows that match the inspection criteria.
Policy: It specifies some policy-level behavior parameters as well as a reusable collection of stateless and stateful rule groups.
Firewall: It enforces the firewall policy's inspection rules to the VPC that the rules are designed to defend. One firewall policy is necessary for each firewall. The firewall also specifies options for stateful traffic filtering and how to log information about your network traffic.

Deployment models for AWS Network Firewall

With AWS Network Firewall, various deployment types are possible. The appropriate model will rely on the requirements and use case. The following models are the most widely used:

Distributed model: Amazon Network Firewall is deployed using a distributed deployment strategy in each VPC that needs security. Each VPC is independently secured, and VPC isolation reduces the explosion radius.
Centralized AWS Network Firewall deployment model: In this for North-South (internet egress and ingress, on-premises) and/or East-West (VPC-to-VPC) traffic, a centralized VPC is used to deploy the AWS Network Firewall. AWS Transport Gateway is a requirement in this model. As a network hub, AWS Transit Gateway streamlines connectivity between VPCs.

Steps walkthrough
Under firewall creation, we will be creating Network Firewall Rule Group and Firewall policy first.

Create Firewall RuleGroup

In this exercise, we will create two sets of rules:

Alerting ICMP traffic
Domain filter Go to VPC > Amazon Network Firewall > Network Firewall rule groups and click on Create Network Firewall rule group

Under Rule group type:
• Select Stateful rule group
• Specify Name
• Set Capacity to 100
• Choose 5-tuple from the Stateful rule group selections.

Choose ICMP as the protocol under "Add rule" and in action select alert; otherwise, leave everything else as is. Then, click "Add rule."

After adding the rule, scroll down and select Create stateful rule. Icmp-alert rule has successfully been created for group.

Create domain filter rule group

To filter/allow particular domains, a rule will now be created:
Go to VPC > Amazon Network Firewall > Network Firewall rule groups and click on Create Network Firewall rule group
Under Rule group type:
• Select Stateful rule group
• Specify Name
• Set Capacity to 100
• Choose Domain list under Stateful rule group options.

Under Domain list:
• Add .amazon.com
• Choose both HTTP and HTTPS under Protocols.
• Under Action select Allow
select Build stateful rule group

We have now created two rules.

Create Firewall Policy

To filter network traffic, you will construct firewall policies in this activity.
Go to VPC > Amazon Network Firewall > Firewall policies and click on Create firewall policy

Under Describe firewall policy:
Specify Name and click on Next

Under Add rule groups:
• Select for all packets, take the same step
• Under Action select Forward to stateful rule groups

Add rule groups by selecting both the domain-allow-centerlized and icmp-centerized rules under Stateful rule group.
Click Next, go over the information, and then select Build firewall policy.

At this point we currently have one firewall policy created

Create Firewall

We will build a firewall in this activity to inspect and filter network traffic.
Go to VPC > Amazon Network Firewall > Firewalls and click on Create firewall
Under Firewall details:
• Specify Name
• Under VPC, select inspection (VPC name, where the firewall
needs to be deployed)
• Under Availability Zone, select respice zones
• Under Subnet select firewall subnet and click on Next
Note: Right now we are going with one AZ deployment

Under Associated firewall policy:
• Select Associate an existing firewall policy and select inspction-firewall-centerlized

We currently have a Firewall that is provisioning. Be patient; it can take a while.

Configure Logging

Let's configure logging while your firewall is being provisioned:
Scroll down to Logging and click on Edit:

In the Firewall logging configuration edit box:
• For Log type select both Alert and flow
• Choose the CloudWatch log group under Log destination for alerts.
/network-firewall-centrlized
• Under Log destination for flows select CloudWatch log group and select /network-firewall-centrlized
and click Save:

At this point, we have a firewall in Ready state
Once firewall is in Ready state, we need to collect firewall endpoint id. These are the endpoints that will serve as targets for traffic directed at your firewall. Next, update the route table. Next, update the routing table adding the endpoint id with the respective availability zone route table.
You can inspect firewall logs after everything is linked and traffic is passing via the network firewall. The CloudWatch logs may take three to six minutes to update.

Reference link: https://catalog.workshops.aws/networkfirewall/en-US/setup/centralmodel/create-anfw/create-firewall-rule-group

Finding unused Amazon EBS volumes using AWS Lambda, EventBridge, and SNS for cost optimization

Rasanpreet Singh — Mon, 06 Mar 2023 04:38:44 +0000

Overview
Cloud cost control is one of the top goals for customers across all sectors and industries. With respect to the AWS EBS storage service, unused resource expenses may be incurred if the lifecycle of volumes is not fully observable. Hence, Amazon EBS volumes that are unused or are forgotten about, add to AWS charges.
In this article, we'll show you how to utilize AWS Lambda, Amazon EventBridge, and AWS SNS to discover EBS volumes that are idle and disconnected from an EC2 instance by receiving alerts through email notifications. This strategy will aid in cost reduction and cost optimization.
In order to list all the unused EBS volumes and send email notifications using SNS topics, we will build a lambda function for this solution. In the following steps, we will establish an Amazon EventBridge rule that will automatically call the lambda function once a week. As a result, we can compile a list of all unused EBS volumes on a weekly basis in a particular AWS region.

Prerequisite
To receive email notifications, we require one subscribed AWS SNS topic. We will utilize the SNS topic ARN in Lambda code.
The Lambda IAM role includes SNS publish, EBS volume describe, list and basic lambda execution permissions.

Steps walkthrough

Create Lambada function
· Visit the Lambda Service Dashboard using the Amazon Management Console. On the Lambda dashboard, select Create Function.
· After that, click Author from Scratch, specify the name of the function, and select Python 3.7 as the runtime. Then pick the lambda service role and select the Create option.

Note: Please ensure that the SNS and EBS permissions policies are associated with the lambda execution role.
· Then open a code editor, begin writing the code

· Enter the following code into the Lambda function with the correct SNS topic ARN and then choose Deploy.

import boto3
def lambda_handler(event, context):
    ec2_client = boto3.client('ec2')
    sns_client = boto3.client('sns')
    volumes = ec2_client.describe_volumes()
    sns_arn = '<SNS Topic ARN>'

    unused_vols = []
    for volume in volumes['Volumes']:
        if len(volume['Attachments']) == 0:
            unused_vols.append(volume['VolumeId'])
            print(volume)


    email_body = "##### Unused EBS Volumes ##### \n"

    for vol in unused_vols:
        email_body = email_body + f"VolumeId = {vol} \n"


    # Send Email

    sns_client.publish(
        TopicArn = sns_arn,
        Subject = 'Unused EBS Volumes List',
        Message = email_body
    )
    print(email_body)

· Now the lambda function is ready for execution.

Create EventBridge Schedule Lambda on Weekly Basis
· Navigate Amazon EventBridge service and open rules. And click on create rule.

· Mention the rule name, select the schedule option, and specify a Cron expression by selecting the cron-based schedule. Here we are using the cron expression which will trigger once a week.

· In the Targets details, select the AWS Lambda option and select our lambda function which we build in the earlier step and then choose Create rule.

· The Lambda function will now automatically get triggered every week to identify the unused EBS volumes and send email alerts using the SNS topic as below.

Conclusion
In this article, we showed you how to receive email notifications about a list of unused EBS volumes so you may check them out for further action and delete them if they're not required to minimize the cost of your monthly Amazon bill.