DEV Community: Deepak Poudel

Automating ENI Failover with AWS Lambda + EventBridge

Deepak Poudel — Tue, 16 Sep 2025 08:22:18 +0000

*Automating ENI Failover with AWS Lambda + EventBridge *

Goal: Automatically detach one or more Elastic Network Interfaces (ENIs) from a primary EC2 instance and attach them to a secondary EC2 instance when the primary transitions to a stopped/terminated state. This guide walks you through the setup entirely from the AWS Management Console.

*Architecture overview *

1.EventBridge receives EC2 Instance State-change Notification for the primary instance (e.g. stopped, terminated).

EventBridge triggers the Lambda function.

3.Lambda calls EC2 APIs to detach the configured ENIs from the primary instance and attach them to the secondary instance.

*Prerequisites *

Primary and secondary EC2 instance IDs.
ENI IDs you want to move (ENIs must be in the same Availability Zone as the target instance).
IAM permissions to create roles, Lambda, and EventBridge rules.

*Step 1 — Create the IAM role for Lambda *

Go to the IAM Console → Roles → Create role.
Select Trusted entity type: AWS service → Use case: Lambda → Next.
Attach the managed policy AWSLambdaBasicExecutionRole (for logging).
**Click Next.
Name the role lambda-eni-mover-role and create it. **\
After creation, open the role and go to the Permissions tab → Add permissions → Create inline policy.
Choose JSON editor and paste:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeNetworkInterfaces",
"ec2:DetachNetworkInterface",
"ec2:AttachNetworkInterface"
],
"Resource": "*"
}
]
}

Save the policy with name LambdaEC2ENIPermissions.

*Step 2 — Create the Lambda function *

Go to the Lambda Console → Create function.
Choose Author from scratch.
Function name: eni-mover
Runtime: Python 3.10
Execution role: Choose Use the existing role and pick lambda-eni-mover-role.
Click the Create function.
Once created, scroll down to the Code Sourc editor. Replace the default code with the provided Python code (see below).
In the Configuration tab → General configuration, set: Timeout: 3 minutes (180 seconds) Memory: 512 MB (adjustable)
In the Configuration tab → Environment variables, add: PRIMARY_INSTANCE = your primary instance ID SECONDARY_INSTANCE = your secondary instance ID SECONDARY_ENIS = comma-separated ENI IDs (e.g., eni-abc123,eni-def456)

`import boto3
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
ec2 = boto3.client('ec2')
PRIMARY_INSTANCE = "INSTANCE_ID_1"
SECONDARY_INSTANCE = "INSTANCE_ID_2"
SECONDARY_ENIS = [
"ENI_ID_1",
"ENI_ID_2"
]
def wait_for_available(eni_id, timeout=180):
start = time.time()
while time.time() - start < timeout:
eni =
ec2.describe_network_interfaces(NetworkInterfaceIds=[eni_id])['NetworkInterfaces
'][0]
if eni['Status'] == 'available':
print(f"ENI {eni_id} is now available.")
return True
time.sleep(3)
raise TimeoutError(f"ENI {eni_id} did not become available in {timeout}
seconds")
def wait_for_in_use(eni_id, timeout=180):
start = time.time()
while time.time() - start < timeout:
eni =
ec2.describe_network_interfaces(NetworkInterfaceIds=[eni_id])['NetworkInterfaces
'][0]
if eni['Status'] == 'in-use':
print(f"ENI {eni_id} is now attached.")
return True
time.sleep(3)
raise TimeoutError(f"ENI {eni_id} did not attach in {timeout} seconds")
def move_eni_to_secondary(eni_id, device_index):
eni =
ec2.describe_network_interfaces(NetworkInterfaceIds=[eni_id])['NetworkInterfaces
'][0]
attachment = eni.get('Attachment')
if attachment:
print(f"Detaching ENI {eni_id} from {attachment['InstanceId']}...")
ec2.detach_network_interface(AttachmentId=attachment['AttachmentId'],
Force=True)
wait_for_available(eni_id)
else:
print(f"ENI {eni_id} already detached.")
print(f"Attaching ENI {eni_id} to secondary at DeviceIndex
{device_index}...")
ec2.attach_network_interface(NetworkInterfaceId=eni_id,
InstanceId=SECONDARY_INSTANCE, DeviceIndex=device_index)
wait_for_in_use(eni_id)
return eni_id
def lambda_handler(event, context):
print("Event received:", event)
detail = event.get("detail", {})
if detail.get("instance-id") != PRIMARY_INSTANCE:
print("Event not for primary instance. Skipping.")
return {"status": "skipped - not primary"}
if detail.get("state") not in ["stopping", "stopped", "shutting-down",
"terminated"]:
print("Instance state not relevant. Skipping.")
return {"status": "skipped - irrelevant state"}

Collect current device indices on secondary

existing_indexes = []
response = ec2.describe_instances(InstanceIds=[SECONDARY_INSTANCE])
for iface in
response['Reservations'][0]['Instances'][0]['NetworkInterfaces']:
existing_indexes.append(iface['Attachment']['DeviceIndex'])

Prepare device indices for each ENI

device_indices = []
next_index = 1
for _ in SECONDARY_ENIS:
while next_index in existing_indexes:
next_index += 1
device_indices.append(next_index)
existing_indexes.append(next_index)
next_index += 1

Move ENIs in parallel

results = []
with ThreadPoolExecutor(max_workers=len(SECONDARY_ENIS)) as executor:
future_to_eni = {executor.submit(move_eni_to_secondary, eni, idx): eni
for eni, idx in zip(SECONDARY_ENIS, device_indices)}
for future in as_completed(future_to_eni):
eni_id = future_to_eni[future]
try:
result = future.result()
print(f"{result} moved successfully.")
results.append(result)
except Exception as e:
print(f"Error moving {eni_id}: {e}")`

10. Click Deploy.
Step 3 — Create the EventBridge Rule

Go to the Amazon EventBridge Console → Rules → Create rule.
Name: eni-mover-primary-stop.
Rule type: Rule with event pattern.
Event pattern → Custom pattern (JSON editor) → paste:

`10. Click Deploy.
Step 3 — Create the EventBridge Rule

Go to the Amazon EventBridge Console → Rules → Create rule.
Name: eni-mover-primary-stop.
Rule type: Rule with event pattern.
Event pattern → Custom pattern (JSON editor) → paste:`
Next, add a Target → choose Lambda function → select eni-mover.
Create the rule.
EventBridge will now automatically invoke the Lambda

aws #awscommunitybuilder #awscommunitynepal #communitybuilder

Public link: https://awsclassstudy.s3.us-east-1.amazonaws.com/Automating+ENI+Failover.pdf

Robust Observability for your AWS Resources with New Relic

Deepak Poudel — Mon, 02 Dec 2024 11:30:45 +0000

Monitoring AWS Resources with New Relic for Enhanced Observability

In this blog, we’ll discuss how your AWS resources can be monitored using a comprehensive monitoring and observability platform to improve application performance, optimize infrastructure usage, and enhance the overall user experience. By leveraging New Relic's suite of tools, you can gain deep insights into your system's behavior, quickly identify and resolve issues, and make data-driven decisions for future enhancements. The primary objective of selecting New Relic as a monitoring platform is to ensure optimal cloud infrastructure performance and resource utilization. It aims to enhance application speed and reliability by identifying and resolving bottlenecks. The platform facilitates streamlined troubleshooting by aggregating and analyzing logs from AWS ECS, Amazon CloudWatch, and other services. Additionally, it enables proactive alerting to notify teams of anomalies and critical incidents while offering insights into user behavior to improve real user experience.

Leveraging AWS and Its Limitations

While AWS Monitoring tools provide infrastructure monitoring, they fall short in several key areas crucial for smooth application performance:

Limited Visibility: AWS CloudWatch primarily focuses on infrastructure metrics, lacking detailed application performance insights such as distributed tracing and code-level troubleshooting.
Siloed Data: Correlating application behavior with underlying infrastructure metrics from CloudWatch can be challenging due to siloed data storage.
Alert Fatigue: CloudWatch's generic alerts can lead to alert fatigue, potentially causing critical issues to be missed.
Limited User Experience (UX) Insights: Understanding how real users experience applications on AWS is difficult with CloudWatch alone.

Key Features of New Relic

Application Performance Monitoring (APM)

New Relic's APM provides comprehensive insights into application performance:
- Detailed Metrics: Monitoring response times, throughput, error rates, and Apdex scores.
- Transaction Tracing: Tracing individual transactions to identify bottlenecks.
- Error Analysis: Automatic capture and analysis of errors for quick resolution.
- Database Monitoring: Monitoring database performance and query execution times.
- Service Maps: Visualizing service dependencies and interactions in real-time.
Infrastructure Monitoring

New Relic Infrastructure offers real-time monitoring for servers and cloud environments:
- Resource Utilization: Tracking CPU, memory, disk I/O, and network usage.
- AWS Integration: Monitoring AWS services like EC2, RDS, S3, and Lambda.
- Health Metrics: Providing health metrics and alerts for uptime and performance.
- Configuration Management: Tracking configuration changes and their performance impact.
- Cluster Monitoring: Monitoring containerized environments like Kubernetes and Docker.
Browser Monitoring / Real User Monitoring (RUM)

Tracks user interactions with your application in real-time:
- Performance Metrics: Measuring page load times and user interaction timings.
- User Sessions: Analyzing user behavior patterns and performance issues.
- Geographical Insights: Identifying performance variations across locations.
- Browser Performance: Tracking across browsers and devices.
- Session Traces: Drilling into individual sessions to identify issues.
Synthetic Monitoring

Simulates user transactions to proactively identify performance issues:
- Scripted Browsers: Creating synthetic scripts for user interactions.
- Global Testing: Testing from multiple geographic locations.
- Performance Benchmarks: Benchmarking against predefined SLAs.
- Alerting: Alerting on performance deviations and downtime.
- Uptime Monitoring: Ensuring application availability with regular checks.
Logs Management

Aggregates and analyzes log data for troubleshooting and optimization:
- Log Aggregation: Collecting logs from applications and infrastructure.
- Search and Query: Performing advanced searches on log data.
- Real-time Streaming: Streaming log data for immediate insights.
- Correlation: Correlating logs with performance metrics for holistic analysis.
- Alerting: Setting alerts for specific log patterns or anomalies.
Dashboards and Alerts

Customizable dashboards and alerting systems to monitor metrics and notify teams:
- Custom Dashboards: Visualizing key metrics and KPIs.
- Pre-built Dashboards: Utilizing pre-built dashboards for common use cases.
- Alerting Policies: Defining alert policies for critical metrics.
- Notification Channels: Integrating with email, Slack, PagerDuty, etc.
- Incident Management: Tracking incidents and resolutions.
Distributed Tracing

Visualizes request flows across services to pinpoint failures:
- Trace Requests: Tracing the journey of requests through services.
- Latency Analysis: Identifying latency at each hop.
- Service Maps: Visualizing service interactions and dependencies.
- Error Tracking: Tracking errors across distributed services.
- Trace Sampling: Managing trace data volume through adaptive sampling.
Mobile Monitoring

Provides insights into mobile application performance:
- Crash Reporting: Automatic capture of application crashes.
- Performance Metrics: Monitoring app launch times and network requests.
- User Interaction Traces: Tracing interactions and their impact on performance.
- Device Insights: Analyzing performance across devices and OS versions.
Serverless Monitoring

Tracks serverless function performance and usage:
- AWS Lambda Integration: Monitoring Lambda functions with real-time metrics.
- Function Tracing: Tracing invocations and dependencies.
- Cost Monitoring: Tracking execution costs and usage.
- Cold Start Analysis: Identifying cold start issues.
- Event Tracking: Monitoring events triggering functions.

How You Can Adopt

Phase 1: Initial Assessment

Identify critical applications and infrastructure components that require monitoring and define performance metrics. This phase involves conducting a thorough assessment to identify critical applications and AWS infrastructure components that require monitoring. Stakeholders will collaborate to define performance metrics and proactive alerting thresholds aligned with business objectives. Leveraging New Relic’s pre-built integrations for AWS services like ECS, Lambda, and RDS aims to streamline integration and enhance visibility into system performance.

Phase 2: Configuration and Integration

Set up New Relic accounts, deploy monitoring agents, and integrate with AWS CloudWatch logs. During this phase, the focus is on setting up New Relic accounts tailored to organizational structure and operational needs. Configuration of monitoring agents across applications and infrastructure will ensure comprehensive data collection. Deployment of browser and synthetic monitoring capabilities will provide real-time insights into user interactions and simulate user journeys for proactive issue detection. Integration with AWS CloudWatch logs will centralize log management for efficient troubleshooting and incident response.

Phase 3: Dashboard and Alert Setup

Develop customized dashboards and set up alerting policies based on predefined thresholds. Customized dashboards within New Relic will be developed in this phase to visualize key metrics and KPIs essential for monitoring application performance, infrastructure health, and user experience. These dashboards will facilitate informed decision-making and enhance operational transparency. Setting up alerting policies based on predefined thresholds will ensure timely notifications via email, Slack, or PagerDuty, enabling swift responses to performance anomalies and security incidents.

Phase 4: Continuous Monitoring and Optimization

Analyze performance data, refine dashboards, and enhance operational efficiency over time. The final phase focuses on establishing a cycle of continuous improvement by analyzing application performance data to identify optimization opportunities. Refinement of dashboards and alerting mechanisms based on usage patterns and feedback will maintain relevance and effectiveness. The goal is to leverage New Relic’s insights to enhance application performance, optimize resource utilization, and strengthen overall operational efficiency over time.

Benefits

Unified View: New Relic eliminates the need for context switching between CloudWatch and separate APM tools by providing a unified view of application performance and infrastructure metrics. This integration simplifies monitoring workflows, enhances efficiency, and facilitates quick correlation of data across different layers of your infrastructure. Improved collaboration among teams further enhances operational effectiveness.
Increased Application Performance: New Relic's advanced APM capabilities, including distributed tracing and code-level profiling, swiftly identify performance bottlenecks that impact application responsiveness. By analyzing detailed metrics and transaction traces, teams can efficiently resolve issues and proactively prevent regressions with synthetic monitoring. This proactive approach ensures a seamless user experience during deployments and under varying workloads.
Reduced Downtime: With New Relic's customizable alerting system and real-time monitoring capabilities, teams receive early warnings about critical metrics and infrastructure health. This proactive monitoring enables prompt mitigation of potential issues before they escalate into outages, ensuring uninterrupted business operations. Enhanced visibility into infrastructure health and centralized log management further accelerates incident resolution, minimizing downtime impact.
Optimized Resource Usage: New Relic provides granular insights into resource consumption across AWS environments, facilitating informed decisions on resource optimization. By identifying and addressing resource bottlenecks, teams optimize infrastructure efficiency and implement cost-saving strategies like AWS Reserved Instances or Auto Scaling. This approach maximizes the value of AWS investments while maintaining optimal performance.
Enhanced User Satisfaction: Improves user experience through RUM insights. Proactively resolving application performance issues with New Relic enhances user experience by ensuring faster response times and improved reliability. Real User Monitoring (RUM) insights into user interactions empower teams to prioritize enhancements that directly impact user satisfaction and retention. Data-driven decisions based on user behavior analytics further optimize user experience, fostering loyalty and engagement.
Proactive Issue Resolution: Customizable alerts for critical performance issues. New Relic's early anomaly detection and comprehensive visibility into application performance enable proactive issue resolution. Customizable alerts notify teams of critical performance issues or potential security threats, enabling swift action to mitigate risks and minimize downtime. Streamlined troubleshooting with centralized log management and distributed tracing ensures quicker identification and resolution of root causes, improving overall incident management efficiency.

Conclusion

Integrating New Relic with AWS enhances monitoring by addressing CloudWatch’s limitations, providing deep insights into application performance, and improving user satisfaction with proactive issue resolution. Start for free today and unlock better observability.

aws #awscommunitybuilder #awscommunitynepal #communitybuilder

Event Driven Shared Drive in AWS

Deepak Poudel — Fri, 15 Nov 2024 10:25:14 +0000

This blog outlines the architecture and setup for Network File Sharing, and Event-Driven Processing using AWS services. The primary components include an Amazon EC2 instance configured as an FTP server with Amazon EFS mounted for shared storage, network file sharing enabled via SAMBA, and event-driven processing handled through AWS Lambda functions triggered by AWS EventBridge. The objective is to create a scalable, secure, and automated environment for file storage, sharing, and processing.

Fig: AWS Architecture Diagram

Create a EC2 instance and mount a NFS Drive

**Mount EFS on the EC2 Instance**

SSH into the EC2 instance and install NFS utilities:
```
`sudo yum install -y amazon-efs-utils`
```
Create a directory for mounting the EFS:
```
`sudo yum install -y amazon-efs-utils`
```

Mount the EFS using the file system ID:

`sudo mount -t efs -o tls fs-XXXXXXXX:/ /mnt/efs`

Add an entry to /etc/fstab to ensure EFS is remounted on reboot:

`echo "fs-XXXXXXXX:/ /mnt/efs efs _netdev,tls 0 0" | sudo tee -a /etc/fstab`

Check if the EFS is successfully mounted:
```
`df -h`
```

Then Setup Samba so that windows devices can directly add network drive

Install SAMBA on the EC2 Instance:

sudo yum install -y samba samba-client samba-common

Backup the default SAMBA configuration file:

sudo cp /etc/samba/smb.conf /etc/samba/smb.conf.bak

Edit the SAMBA Configuration:

sudo nano /etc/samba/smb.conf

The following settings are configured under the [global] section:

[global]
workgroup = WORKGROUP
server string = Samba Server
netbios name = ftp-server
security = user
map to guest = bad user

Following share configuration is added to allow Windows clients to access the EFS directory:

[EFS_Share]
path = /mnt/efs
browseable = yes
writable = yes
guest ok = yes
create mask = 0755
directory mask = 0755

Start the SAMBA services to apply the configuration:
```
sudo systemctl start smb
sudo systemctl start nmb
```
Enable SAMBA services to start on boot:
```
sudo systemctl enable smb
```

Now when the file is uploaded to the FTP server or on via Samba, We need a REPL that checks for changes and sends them for processing \
\
#!/bin/bash

# Set variables
SOURCE_DIR="/mnt/efs/fs1"
S3_BUCKET="s3://backup-efs-ftp-bucketffa/"
LOG_FILE="/home/ec2-user/upload_to_s3.log"
DEBOUNCE_DELAY=30  # Delay in seconds for file stability check

# Function to log messages
log_message() {
   echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE"
}

# Function to check if file size is stable and not locked
is_file_stable() {
   local file="$1"
   local prev_size=$(stat -c%s "$file")
   log_message "Initial size of '$file': $prev_size bytes"


   # Sleep for the debounce delay
   sleep "$DEBOUNCE_DELAY"


   local new_size=$(stat -c%s "$file")
   log_message "Size of '$file' after sleep: $new_size bytes"


   # Check if the file size is stable
   if [ "$prev_size" -eq "$new_size" ]; then
       log_message "Size of '$file' after sleep didn't changed."
       # Now check if the file is locked
       if lsof "$file" &>/dev/null; then
           log_message "File '$file' is locked after stability check."
           return 1  # File is locked
       else
           log_message "File '$file' is stable and not locked."
           return 0  # File is stable and not locked
       fi
   else
       log_message "File '$file' size changed during stability check."
       return 1  # File is still changing
   fi
}

# Function to upload file to S3
upload_to_s3() {
   local file="$1"
   local full_path="$SOURCE_DIR/$file"


   # Check if the file exists
   if [ ! -f "$full_path" ]; then
       log_message "File '$full_path' does not exist. Skipping upload."
       return
   fi


   # Ensure the file size is stable and not locked
   if ! is_file_stable "$full_path"; then
       log_message "File '$full_path' is still changing or locked. Delaying processing."
       return
   fi


   # Create destination path for S3
   local s3_path="${S3_BUCKET}${file}"

   # Upload file to S3
   log_message "Attempting to upload '$full_path' to S3 path '$s3_path'..."
   if aws s3 cp "$full_path" "$s3_path" --acl bucket-owner-full-control; then
       log_message "Successfully uploaded '$file' to S3"
   else
       log_message "Failed to upload '$file' to S3. Error code: $?"
   fi
}

# Main loop to monitor directory recursively
log_message "Starting to monitor '$SOURCE_DIR' for new files..."
inotifywait -m -r --format '%w%f' -e close_write -e moved_to "$SOURCE_DIR" |
while read -r full_path; do
   # Clean up the filename to remove unwanted characters
   clean_filename=$(basename "$full_path")

   # Debugging information
   echo "Detected full path: '$full_path'"
   echo "Cleaned filename: '$clean_filename'"

   # Log detected file
   log_message "Detected new file: '$full_path'"

   # Ignore temporary or partial files
   if [[ "$clean_filename" != .* ]] && [[ "$clean_filename" != *.part ]] && [[ "$clean_filename" != *.tmp ]]; then
       # Wait for the debounce delay before uploading
       if is_file_stable "$full_path"; then
           upload_to_s3 "${full_path#$SOURCE_DIR/}"  # Remove SOURCE_DIR from the path
       else
           log_message "File '$full_path' is still locked or changing. Ignoring this upload attempt."
       fi
   else
       log_message "Ignoring temporary or partial file: '$full_path'"
   fi
done

Now Let’s Move Forward to Event Driven Architecture.

In our case let’s unzip uploaded files if they are zipped.

This is the code for our triggering AWS Lambda for each file upload. If the uploaded file is

import json
import boto3
from handler_run_task import run_ecs_task

s3 = boto3.client('s3')

def lambda_handler(event, context):
    try:
        # Get bucket name and object key from the event
        source_bucket = event['Records'][0]['s3']['bucket']['name']
        object_key = event['Records'][0]['s3']['object']['key']

        # Define ECS cluster and task details
        cluster_name = 'unzip-test-cluster'  # Replace with your ECS cluster name
        task_family = 'ple-family'  # Replace with your ECS task family
        container_name = 'test-container'  # Replace with your container name

        # Define the overrides for the ECS task
        overrides = {
            'environment': [
                {
                    "name": "BUCKET_NAME",
                    "value": source_bucket
                },
                {
                    "name": "KEY",
                    "value": object_key
                }
            ]
        }

        # Run ECS Task
        ecs_response = run_ecs_task(
            cluster_name,
            task_family,
            container_name,
            overrides,
            source_bucket,
            object_key
        )

        return {
            'statusCode': 200,
            'body': json.dumps('ECS task triggered successfully!')
        }

    except Exception as e:
        print(f"Error triggering ECS task: {e}")
        return {
            'statusCode': 500,
            'body': json.dumps(f"Error triggering ECS task: {str(e)}")
        }

**Let’s Create a handler when the previous lambda gives, json with files to process. The json will contain keys and bucket from which to unzip and the image to unzip the files**

ECS Task is created by the lambda function and is able to handle the unzipping of files. If an unzipped file is found, the ECS task unzips the file and uploads the extracted contents back to a target S3 bucket. If the upload fails, the file is routed to a Dead Letter Queue (DLQ). Non-ZIP files are directly copied to the target bucket.

ECS Configuration:

Cluster Name: unzip-test-cluster
Task Family: ple-family
Container: test-container
Launch Type: Fargate

Task Execution:

Retrieves the latest task definition for the given family.
Executes the task with environment variables passed by the Lambda function.

Handler_run_task.py

import boto3

def run_ecs_task(cluster_name, task_family, container_name, overrides, source_bucket, object_key):
    ecs_client = boto3.client('ecs')

    # Get the latest task definition for the given family
    response = ecs_client.list_task_definitions(
        familyPrefix=task_family,
        sort='DESC',
        maxResults=1
    )

    latest_task_definition = response['taskDefinitionArns'][0]
    print("Printing Latest task def")
    print(latest_task_definition)

    # Run the ECS task with the latest task definition
    response = ecs_client.run_task(
        cluster=cluster_name,
        taskDefinition=latest_task_definition,
        overrides={
            'containerOverrides': [
                {
                    'name': container_name,
                    # 'cpu': overrides.get('cpu', 512),  # Default CPU to 512
                    # 'memory': overrides.get('memory', 1024),  # Default memory to 1024 MiB
                    'environment': overrides.get('environment', [])
                }
            ]
        },
        networkConfiguration={
            'awsvpcConfiguration': {
                'subnets': ['subnet-089f9162bd2913570', 'subnet-05591da28974513ee', 'subnet-0732585a95fcd1b64'],  # Replace with your subnet ID
                'assignPublicIp': 'ENABLED'  # or 'DISABLED' depending on your network setup
            }
        },
        launchType='FARGATE',  # Or 'EC2', depending on your setup
        count=1
    )

    return response

**Now let’s create a service linked role.**

An IAM role is created to grant the Lambda function and ECS tasks the necessary permissions to access other AWS resources, such as S3, ECS, and CloudWatch Logs.

Key Policies:

Log creation and event publishing to CloudWatch
S3 bucket access (GetObject, ListBucket, PutObject)
ECS task execution and description

IAM role passing

{
"Version": "2012-10-17",
"Statement": [
{
    "Effect": "Allow",
    "Action": "logs:CreateLogGroup",
    "Resource": "arn:aws:logs:<aws_region>:<account_id>:*"
},
{
    "Effect": "Allow",
    "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
    ],
    "Resource": "arn:aws:logs:<aws_region>:<account_id>:log-group:/aws/lambda/<lambda_function_name>:*"
},
{
    "Effect": "Allow",
    "Action": [
        "s3:GetObject",
        "s3:ListBucket"
    ],
    "Resource": [
        "arn:aws:s3:::<source_bucket_name>",
        "arn:aws:s3:::<source_bucket_name>/*"
    ]
},
{
    "Effect": "Allow",
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::<target_bucket_name>/*"
},
{
    "Effect": "Allow",
    "Action": "iam:PassRole",
    "Resource": "arn:aws:iam::<account_id>:role/*"
},
{
    "Effect": "Allow",
    "Action": [
        "ecs:RegisterTaskDefinition",
        "ecs:DescribeTaskDefinition"
    ],
    "Resource": "*"
}
]
}

**Dead-letter Queue (DLQ) Mechanism**

The DLQ captures and logs events where the ECS task fails to process the uploaded file correctly. This mechanism ensures that errors are captured and stored for subsequent analysis or reprocessing.

Failure Handling

Condition: A failure is identified if the ECS task returns an HTTP status code other than 200.
DLQ Logging: The failed event, including file details and error messages, is sent to the DLQ. The DLQ serves as a reliable storage for these failed events, ensuring no data is lost.

Now let’s create an ecs image \
\
Here goes the unzip file \
\
#!/bin/bash

# Environment variables
SRC_BUCKET_NAME="$BUCKET_NAME"
SRC_BUCKET_KEY="$KEY"
DEST_BUCKET_NAME="backup-efs-ftp-ple-unzipped"
DLQ_URL="https://sqs.us-east-1.amazonaws.com/654654543848/unzip-failed-processing-queue"
OUTPUT_DIR="./output"
LOCAL_FILE_PATH="./$(basename "$SRC_BUCKET_KEY")"

# Function to log messages
log_message() {
   echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
}

# Function to download a file from S3
download_file() {
   log_message "Downloading $SRC_BUCKET_KEY from $SRC_BUCKET_NAME"
   aws s3 cp "s3://$SRC_BUCKET_NAME/$SRC_BUCKET_KEY" "$LOCAL_FILE_PATH"
   if [ $? -ne 0 ]; then
       log_message "Error downloading file from S3"
       send_to_dlq "Error downloading file from S3"
       exit 1
   fi
}

# Function to upload a file to S3
upload_to_s3() {
   local file_path="$1"
   local s3_key="$2"
   log_message "Uploading $file_path to s3://$DEST_BUCKET_NAME/$s3_key"
   aws s3 cp "$file_path" "s3://$DEST_BUCKET_NAME/$s3_key" --acl bucket-owner-full-control
   if [ $? -ne 0 ]; then
       log_message "Failed to upload $file_path to S3"
       send_to_dlq "Failed to upload $file_path to S3"
       exit 1
   fi
   invoke_load_balancer "$s3_key"
}

# Function to send a message to the DLQ
send_to_dlq() {
   local message="$1"
   log_message "Sending message to DLQ: $message"
   aws sqs send-message --queue-url "$DLQ_URL" --message-body "$message"
   if [ $? -ne 0 ]; then
       log_message "Failed to send message to DLQ"
       exit 1
   fi
}

# Function to invoke load balancer
invoke_load_balancer() {
   local s3_key="$1"
   log_message "Invoking load balancer for $s3_key"
   local payload=$(jq -n \
       --arg bucket "$DEST_BUCKET_NAME" \
       --arg key "$s3_key" \
       '{bucket: $bucket, key: $key, filePath: $key}')
   local response=$(curl -s -X POST "https://asdlb.mydomain.com/companyx/gateway/listenFTPWebhook" \
       -H "Content-Type: application/json" \
       -d "$payload")

   local status_code=$(echo "$response" | jq -r '.status')
   if [ "$status_code" != "200" ]; then
       log_message "Load balancer invocation failed with status code $status_code"
       send_to_dlq "Load balancer invocation failed with status code $status_code"
   else
       log_message "Load balancer invocation successful"
   fi
}

# Function to extract ZIP files
extract_and_process_files() {
   log_message "Extracting ZIP file $LOCAL_FILE_PATH"
   mkdir -p "$OUTPUT_DIR"
   unzip -o "$LOCAL_FILE_PATH" -d "$OUTPUT_DIR"
   if [ $? -ne 0 ]; then
       log_message "Failed to extract ZIP file"
       send_to_dlq "Failed to extract ZIP file"
       exit 1
   fi

   for file in "$OUTPUT_DIR"/*; do
       local s3_key=$(dirname "$SRC_BUCKET_KEY")/$(basename "$file")
       log_message "Processing file $file"
       upload_to_s3 "$file" "$s3_key"
   done
}

# Main process
main() {
   log_message "Starting processing for $SRC_BUCKET_KEY from $SRC_BUCKET_NAME"
   download_file

   if [[ "$LOCAL_FILE_PATH" == *.zip ]]; then
       extract_and_process_files
   else
       local s3_key=$(dirname "$SRC_BUCKET_KEY")/$(basename "$LOCAL_FILE_PATH")
       upload_to_s3 "$LOCAL_FILE_PATH" "$s3_key"
   fi

   log_message "Processing complete"
}

main

And here’s the Docker File for the same \
\
# Use the official lightweight Debian image as the base

FROM debian:bookworm-slim

# Set the working directory
WORKDIR /usr/src/app

# Set non-interactive mode to avoid prompts during package installation
ENV DEBIAN_FRONTEND=noninteractive

# Install necessary tools with cache cleanup
RUN apt-get update && \
   apt-get install -y --no-install-recommends \
   bash \
   curl \
   unzip \
   awscli \
   apt-get clean && \
   rm -rf /var/lib/apt/lists/*

# Copy your shell script into the container
COPY unzip.sh /usr/src/app/

# Make the shell script executable
RUN chmod +x /usr/src/app/unzip.sh

# Set environment variables (can be overridden at runtime)
ARG BUCKET_NAME
ARG KEY

# Command to run your shell script
CMD ["/usr/src/app/unzip.sh"]

Build and Push the Docker Image to ECR and update the code.

Conclusion

We have Shared Samba Server for windows users with Event Driven on demand unzip functionality when a user uploads file to the Drive.

aws #awscommunitybuuilder #awscommunitynepal #communitybuilder

Complete guide to protect Amazon EBS Snapshots from accidental deletion using AWS Recycle Bin

Deepak Poudel — Tue, 12 Mar 2024 06:58:37 +0000

Have you ever been wondering about or facing the problem of accidental deletion of Amazon EBS Snapshots and AMI?
Don’t worry; the AWS Recycle Bin is the solution to such a problem.
Normally, we use EBS snapshots to back up the data on our Amazon EBS volumes. The snapshots are very useful, mainly in disaster recovery, data migration, and backup compliance. But in some cases, those snapshots can be accidentally deleted, which can cause a huge loss of data. For that reason, in this blog, I will be guiding you to use Amazon Recycle Bin.
Let’s get started from the very beginning.
Pre-requisite: Before starting the steps to configure the recycle bin, we must have at least one EBS volume. That EBS volume can be independent of EC2 or attached to an EC2 instance.
Steps
Step1: Configuration of EBS snapshot
First of all, let us create an EBS snapshot from the EBS volume. Navigate to the EC2 service. In the left navbar, click on "Volumes.”.

In the volumes section, we can see a list of existing EBS volumes.

Select the EBS volume using the checkbox on the left side of the volume name.

Now, we need to create an EBS snapshot for the volume above. For that, click on “Action” and then “Create snapshot.”.

Provide the details for the EBS snapshot by filling out the Description and Tags section. Multiple tags can also be provided as per requirement. We will be using the tag while configuring the recycle bin. After that, create the EBS snapshot.

A message is shown after the successful creation of a snapshot with its unique ID.

Step2: Configuration of Recycle Bin
Search for Recycle Bin in the search bar and select that service.

In the recycle bin, we must create a retention rule to protect the snapshot. Click on “Click retention rule.”

In the rule details, provide the retention rule name so that we can identify the rule properly. Also, provide a relevant description of the rule.
In the resource type, select EBS Snapshots. To apply the retention rule to all EBS snapshots, select “Apply to all resources.”. In this blog, I will be applying the retention rule to only one snapshot. Select the relevant tags for the snapshots, which we have to retain, and click on "Add.”. In the retention period, we can choose the time period for which the resources can be recovered after deletion.The minimum time can be 1 day, and the maximum can be 365 days.

After configuring all the details, click on “Create retention rule” at the bottom.

The retention rule that we configured can be seen in the home section of the recycle bin.

The retention rule has one feature for rule lock. Select the rule and click on "Action,” then “Edit retention rule lock.”

In the rule lock setting, we can configure it to prevent the retention rule itself from being accidentally or maliciously updated or deleted. There are two options: “unlock” and “lock.”. If we select Unlock, the rule can be modified and deleted. But if we select lock, then the rule can’t be modified or deleted until it is unlocked and the specified delay period has expired.

Step3: Checking the retention
Now, we need to test the use of the recycle bin by deleting the snapshot. Navigate to the EBS snapshots section by searching “snapshots” in the search bar.

Select the snapshot and click on "Actions,” then “Delete snapshot.”

After successfully deleting the snapshot, we can see that the snapshot no longer exists. This deletion can be done accidentally in some cases. Click on “Recycle Bin” in the top right. This will redirect to the Recycle Bin service.

Select “Resources” just below the recycle bin on the left side.

In the resources section we can see the EBS snapshot that we deleted earlier. It also shows the bin entry date and bin exit date.

To recover the snapshot, select the snapshot and click on “Recover” in the top right section.
Now, click “Recover resources” to retain the snapshot volume

Navigate to the EBS snapshot section where, we can see the recovered snapshot.

In this way we have successfully used Recycle Bin to recover EBS snapshot from accidental deletion. In the similar manner it can also be used to recover Amazon Machine Image (AMI).

Streamlining Containerized Web Application Deployment: An All-Inclusive Guide Using AWS ECS, ECR, and CodePipeline

Deepak Poudel — Thu, 29 Feb 2024 17:29:45 +0000

In today’s world of technology web applications have become a major part of digital experience. They provide a complete and dynamic solution for various needs, ranging from communication and collaboration to e-commerce and entertainment. To ensure portability, efficiency, scalability, and the ability to provide consistent operation across diverse environments, containerization is required. In this blog, I will guide about streamlining containerized web application deployment using AWS services such as ECS, and ECR along with CodePipeline to accommodate continuous change requirements.
First of all, let us understand the use case of services that will be used for this project.

CodeCommit- It is a version control service, which will be used to store code, documents etc.

CodeBuild- It is used to compile source code, runs unit test and produce artifact.

CodePipeline- Used to quickly model and configure the different stages of software release process and automate the software change continuously.

IAM roles- Used to delegate access to AWS resources.
ECR (Elastic Container Registry)- Used to securely store, manage docker containers images.

ECS (Elastic Container Service)- Used to easily deploy, manage and scale docker container.

AWS Fargate- Used with ECS to run containers without having to manage servers.

Let us jump directly to the set of steps
Steps:
Step1:
In the very first, we will configure IAM roles with the permission policy required in this project for AWS services. To create IAM roles, IAM is used. Permission policies required for this project are:

AmazonEC2ContainerRegistryPowerUser
AWSCodeCommitReadOnly
CloudWatchLogsFullAccess
Attach those policies to the role.

Step 2: We need to push our code to a repository for which code commit will be used. Search for CodeCommit in the AWS console and click in “Create Repository”. Repository name must be provided for CodeCommit. Descriptions, tag and CodeGuru reviewer are optional.

After creating git repository, it provides different way of connection such as HTTPS, SSH, HTTPS(GRC). We will be using HTTPS option to clone the repository so that we can push code to CodeCommit.
To clone the repository, use the code: git clone
In the local system where the web app code is available clone the repository and push the code.
Code to create branch main- git checkout -b main
Add files: - git add .
Commit - git commit -m
Push - git push origin main
Content of Docker file used in this project.

FROM nginx
WORKDIR /app
COPY . /usr/share/nginx/html
EXPOSE 80
buildspec.yml file is also required which will be configured later.

Step3:
Now, we will set up ECR. Visibility setting can be choosen between private and public. In this project we will be using private repository. A concise name which can be identified by developer should be provided in repository name.

After creating a container repository, it can be shown listed in private along with its URI.

Click on Repository name to view the push commands.

Step 4:
Now, we will set up CodeBuild with source as CodeCommit. CodeBuild will extract code from the CodeCommit repository that we created earlier. Reference type can be selected as branch with relevant name, in our case “main”.

We will be using default environment configuration. CodeBuild also requires compute medium, for which we will use EC2. We don’t need to configure EC2. To execute operation of CodeBuild permission is required which is provided by IAM role created earlier.
buildspec.yml file will be used to build commands which will be provided through CodeCommit.
The configuration of buidlspec.yml file is given as below.

version: 0.2
phases:
  install:
    runtime-versions:
      nodejs: latest
    commands:
      - nohup /usr/local/bin/dockerd --host=unix:///var/run/docker.sock --host=tcp://127.0.0.1:2375 --storage-driver=overlay2 &
      - timeout 15 sh -c "until docker info; do echo .; sleep 1; done"
  pre_build:
    commands:
      - echo log in to Amazon ECR...
      - <provide way to Retrieve an authentication token and authenticate your Docker client to your registry using push commands>
      - REPOSITORY_URI=<replace with the uri of ECR>
      - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7
      - IMAGE_TAG=${COMMIT_HASH:=latest}
  build:
    commands:
      - echo Build started on `date`
      - echo Building the Docker image.
      - docker build -t $REPOSITORY_URI:latest .
      - docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG
  post_build:
    commands:
      - echo Build completed on `date`
      - docker push $REPOSITORY_URI:latest
      - docker push $REPOSITORY_URI:$IMAGE_TAG
      - echo write definitions file...
      -printf'[{"name":"blog-code-pipeline","imageUri":"%s"}]' $REPOSITORY_URI:$IMAGE_TAG > imagedefinitions.json
artifacts:
  files: imagedefinitions.json

This buildspec file defines the steps for building and pushing docker image to ECR. Major phases for this file are.
Phases:
Install: For our source code to run we need to install Node.js latest version and also start Docker daemon.
Pre_build: Authenticate Amazon ECR and setting variable for Docker image repository along with determining the image tag.
Build: Commands to build Docker image by specifying ECR repository URI and tag.
Post_build: Push Docker image to ECR and create definition file. Pushing definition file to artifact store.
Commands:
Commands for starting Docker daemon, authenticating ECR, building Docker image and pushing that image to ECR, creating the definitions file are used in the buildspec.yml file above.
This buildspec file is used to automate the process of building a Docker image from source code in CodeCommit and tagging it with version, and pushing the Docker image to ECR. The resulting image information will be stored in artifact(imagedefinitions.json) file, for further deployment in ECS.
Note the name “blog-code-pipeline” which we have used.
After writing the buildspec.yml push it to the CodeCommit repository using the same process that we did earlier.
To check whether our setup works well we will run CodeBuild and see the phase details as.

Step 5- In this step we will configure ECS to run the docker container from registry.
First of all, cluster should be made. This will be used to organize container instances regionally so that task requests may be executed on them. Cluster name must be provided along with infrastructure configuration. AWS Fargate serverless will be used in this project. Click on the “create” button in the bottom. The cluster will be made through CloudFormation. It may take few minutes to set up. Progress can be tracked in CloudFormation

Configure task definition- Application's blueprint is found in the task definition. The parameters and one or more containers that make up application are described using it. After creating the cluster, click on the “Task definitions” available in the left navbar.

We can also create task definition using JSON, but for simplicity propose click on “Create new task definition”. Provide the task definition family name and infrastructure to run the task. We will be using default environment setup with Fargate.

We will use the default OS architecture and keep the task size to 1vCPU and 3GB memory.

In the container section give the container name and image URI of repository. Container name is configured as per buildsepc.yml “blog-code-pipeline”. For port mapping we will use port 80, as per our Docker file. We can customize the other option, but for this sample project leave other section as default and click on “create” button at last

Now, click on the cluster that we created earlier and go to the services section. In order to execute and manage a given number of instances of a task description concurrently in an Amazon ECS cluster, we must build services.

The cluster name will be selected by default. We will use Fargate as capacity provider keeping the platform version latest.

In the deployment configuration section service and provide family name along with service name. We will leave other setting as default.

Step 6:
Set up CodePipeline
At last CodePipeline must be configured so that when code is pushed to CodeCommit it trigger CodeBuild and then ECS for deploying changes without requiring manual intervention.
Name should be provided for the pipeline to be identified.
We can create new service role in Code Commit.

The source will be CodeCommit which contains code that we push from our local system. It must be configured according of CodeCommit repository, as our previous configuration.

In the build phase, build provider must be selected and relevant name must be provided

In the last deploy provider must be provided which will be used for deploying the containerized web app. ECS will be used as build provider which we configure earlier. Provide.

the cluster name ,service name and image definitions file accordingly.
After clicking on “Release change” of CodePipeline it will first get the code form CodeCommit and run CodeBuild and deploy using ECS. It may take some time to complete the whole process.

Now, to see the output go to ECS. In the task section of the cluster that we created. We can see public IP address for the hosted web app.

Copy and paste that IP address in the browser where we can see our website is hosted.

Now, if we make any change to our source code of web app and push it to CodeCommit, it will automatically run CodePipeline and the change will be deployed in ECS automatically.
In this way we have successfully deployed our containerized web app using ECS, ECR and CodePipeline.

S3 File Analysis with AWS Lambda: Counting Words and SNS Notifications"

Deepak Poudel — Sat, 10 Feb 2024 08:11:15 +0000

In today's fast-paced world, automation is the key to efficiency. AWS Lambda, one of the popular serverless computing service, allows you to run code without provisioning or managing servers. In this tutorial, I'll walk you through how to leverage AWS Lambda to automatically count the number of words in a file uploaded to an S3 bucket and then send the word count in an email via Amazon SNS (Simple Notification Service). This tool can be valuable for various applications such as content analysis, document processing, and more.

Prerequisites
Before directly diving into the implementation, make sure you have the following prerequisites in place:

● An AWS account with appropriate permissions to create Lambda functions, S3 buckets, and SNS topics.

Step 1: Log into the AWS Management Console and navigate the S3 service.

Step 2: Create a Bucket:
Once you're in the S3 dashboard, click the "Create bucket" button.

Step 3: Configure Bucket Settings:

● Bucket Name: Choose a globally unique name for your S3 bucket. Bucket names must be unique across all of AWS, so picking a name that someone else has yet to use is essential.

● Region: Select the favorable AWS region where you want to create the bucket. Choose a region that is geographically close to your intended users or applications for better performance.

● Configure options such as versioning, logging, and tags as needed: Depending on your specific use case, you can enable features like versioning to keep multiple versions of files or configure logging to track bucket activity.
● Set Permissions: By default, the bucket is private, meaning only the AWS account that created it has access. If you want to grant public access or specific permissions to other AWS accounts or IAM users, you can configure bucket policies, access control lists (ACLs), or IAM policies.

● Review and Create: After configuring the settings for your S3 bucket, review your choices to ensure they are correct. Double-check the bucket name for uniqueness.
● Click the "Create bucket" button to create the S3 bucket.

Step 4: Create an IAM role to permit the Lambda function to access Amazon S3 and Amazon SNS:

● Create a New Role:
➔ In the IAM dashboard, click "Roles" in the left sidebar.

➔ Now click the "Create role" button to create a new role with required permission policies.

● Select Type of Trusted Entity:

➔ For the trusted entity type, select "AWS service" since you're creating this role for AWS Lambda.

➔ In the use case options, choose "Lambda
● Attach Permissions Policies:

In the "Permissions" step, you can attach policies defining the role's actions. Search and select the following policies:

➔ AWSLambdaBasicExecutionRole: This policy lets your Lambda function write logs to CloudWatch Logs.
➔ AmazonSNSFullAccess: This policy provides full access to Amazon SNS, allowing your Lambda function to publish messages to SNS topics.
➔ AmazonS3FullAccess: This policy provides full access to Amazon S3, allowing your Lambda function to read from and write to S3 buckets.
You can search for these policies in the search box and attach them individually.

● Name and Create Role:

➔ Give your role a name. In this case, you can name it "wordCounterRoleforlambda" or choose any other desired name.

➔ Add a description to help you remember the purpose of this role.
➔ Click the "Create role" button to create the role.

Step 5: Create a Simple Notification Service (SNS) topic.
● Navigate to SNS (Simple Notification Service):

➔ Create a New Topic: In the SNS dashboard, click the "Create topic" button to create a new SNS topic.

➔ Configure Topic Details: Choose a type to Standard and Provide a name for your topic.

➔ Click on Create Topic

➔ Create Subscription: With your topic selected, click the "Create subscription" button to set up a new subscription.

➔ Choose a Protocol and Endpoint and Click Create Subscription

➔ You will receive an email shortly to confirm your Subscription

➔ Click on Confirm Subscription

Step 6: Create a Lambda Function
● Go to the AWS Lambda console.
● Click "Create function" and choose "Author from scratch” as shown below.

● Give your function a name, and choose the runtime (e.g., Python 3.7)

● Change the default execution role and select the existing role that we created earlier and click on create Function

● Add a trigger to the lambda function

➔ In Trigger Configuration, Choose S3 , and the bucket we created earlier and click on Add

➔ In lambda_handler.py write a following code

`import boto3
import os
import json

def lambda_handler(event, context):

# Find the topic ARN from the environment variables.

TOPIC_ARN = os.environ['topicARN']
print("Topic ARN =", TOPIC_ARN)

# Create an S3 client and find the S3 bucket name and file name (object key) from the event object.

s3Client = boto3.resource('s3')

record = event['Records'][0]
bucketName = record['s3']['bucket']['name']
print("bucketName =", bucketName)
objectKey = record['s3']['object']['key']
print("objectKey =", objectKey)

# Read the contents of the file.

textFile = s3Client.Object(bucketName, objectKey)
fileContent = textFile.get()['Body'].read()

print("fileContent =", fileContent)

# Count the number of words in given file.

wordCount = len(fileContent.split())
print('Number of words in text file:', wordCount)

# Create an SNS client, and format and publish a message containing the word count to the topic.

snsClient = boto3.client('sns')
message =  'The word count in the file ' + objectKey + ' is ' + str(wordCount) + '.'

response = snsClient.publish(
    TopicArn = TOPIC_ARN,
    Subject = 'Word Count Result',
    Message = message
)

# Return a successful function execution message.

return {
    'statusCode': 200,
    'body': json.dumps('File successfully processed by wordCounter Lambda function')
}`

This AWS Lambda function, when triggered by an S3 event, retrieves the file uploaded to an S3 bucket, counts the number of words in that file, and publishes the word count as a message to an Amazon SNS topic. The Lambda function uses the Boto3 library to interact with AWS services. It starts by extracting the S3 bucket name and object key from the event object, then reads the content of the file from S3. After counting the words, it formats a message and publishes it to the specified SNS topic. Finally, it returns a successful execution message. This code is designed to automate the word-counting process and notify subscribers via SNS when new files are uploaded to the S3 bucket.

● Click on Configuration and Set the Environment variable

➔ Navigate to the sns Topic that we created earlier and copy ARN

➔ Set the Environment Variable

● On Lambda Dashboard, click on Test

● Configure Test Event
➔ Give Event Name
➔ Choose S3-put on the template

➔ Update S3 bucket name, ARN, Principal, and Key on Template

{ "Records": [ { "eventVersion": "2.0", "eventSource": "aws:s3", "awsRegion": "us-east-1", "eventTime": "1970-01-01T00:00:00.000Z", "eventName": "ObjectCreated:Put", "userIdentity": { "principalId": "EXAMPLE" }, "requestParameters": { "sourceIPAddress": "127.0.0.1" }, "responseElements": { "x-amz-request-id": "EXAMPLE123456789", "x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH" }, "s3": { "s3SchemaVersion": "1.0", "configurationId": "testConfigRule", "bucket": { "name": "deepakwordcountbucket", "ownerIdentity": { "principalId": "*" }, "arn": "arn:aws:s3:::deepakwordcountbucket" }, "object": { "key": "word.txt", "size": 1024, "eTag": "0123456789abcdef0123456789abcdef", "sequencer": "0A1B2C3D4E5F678901" } } } ] }

Step 7: Create and Upload a file in amazon s3 bucket that we created earlier
● Create a new file and write some contents

Note that there are 6 words in this file

● Upload this file to amazon s3 bucket (deepakwordcountbucket)

● Click on deploy in lambda function

● You will get an email which will tell you the number of words in the file you uploaded to s3 bucket

Conclusion
In this tutorial, I've shown you how to automate word counting on files uploaded to an S3 bucket using AWS Lambda and send the word count in an email notification via Amazon SNS. This automation can be a valuable addition to various data processing and content analysis workflows, saving you time and effort while keeping you informed about the contents of uploaded files. Explore further customization and integration options to suit your specific use cases.

How AWS saved me a lot of headache in my job

Deepak Poudel — Mon, 24 Apr 2023 17:20:35 +0000

I recently deployed my first solo project for my job at Digo and I’ve never been more grateful for AWS.

Digo is a business management platform where automation is one of its core features. Internally, we call these automations “workflows”. Since Digo has been growing recently and more demanding customers are using our system, we were noticing huge CPU and memory spikes in our server due to workflows. Our server was a monolith and it was time to implement a new workflow engine and move it to a separate service.
Implementation
This was my first independent feature and I was very excited and also a bit nervous! I made sure not to overlook anything. It took weeks to design, discuss and redesign the new workflow engine to finally come upon a performant solution that met all our business needs. I learned a lot from my mentor during this process.

I had a solid view of the feature and how I was going to implement it. But a thought was creeping behind me: How am I going to deploy it?
Deployment Problems
I had very little experience with deploying software to production. All of our infrastructure is deployed in AWS. During my dev cycles, I spin up a free tier ec2 instance early testing and progress demo. I am well familiar with linux as I had done a ground up installation of arch linux in my previous laptop. But deploying softwares in linux and managing it as a server was an entirely new topic for me. I somehow ran the workflow engine server in ec2 with the help of multiple articles on the internet. But I hated every time I had to make new deployments. Looking back my problems came to be because:
I didn’t have deployment automation. Every time I had to deploy, I’d ssh into ec2, pull from master and run the server.
My servers kept crashing. Initially, I didn’t use any process managers like pm2, docker etc. So every time my server crashed, I’d ssh into ec2 and restart the server. Later, I switched to pm2 and it saved me from the need to do this.
I didn’t have a proper logging library. So I had to look through linux log files when my process crashed.

Deploying features for testing during deployment has many benefits. It is good for the feature itself because we get continuous feedback early in the development cycle. It is good for QA, managers and the rest of the team because they can test the workflow engine and get exposed to its features. But, it sucked for me because of my little experience with deployment. I wish I had spent some time learning the basics of DevOps and AWS early. But I decided to focus more of my time on implementing the new workflow engine than learning DevOps.
Resolving Issues with AWS services
My development was over and the feature was reaching release. It was finally time to learn DevOps. I had knowledge transfer sessions with my mentor. I also read more articles and explored AWS. Because I already had such a terrible experience once, I realized the importance of DevOps. I took courses at the AWS academy as well.

Some weeks before production release, I addressed the problems I faced previously. These were all the different AWS services I used:
Docker and Amazon Elastic Container Service (Amazon ECS) to delegate all my server management responsibilities. I didn’t have to worry about managing linux servers, scaling my infrastructure, server availability, etc because Amazon ECS handled everything for me.
Task definition files for ECS to define my infrastructure blueprint for different environments (staging, production, etc)
Cloudwatch logs to stream my container’s logs.
Amazon DynamoDB to delegate all my database management responsibilities.
Amazon parameter store for managing my environment secrets

With these in place, I didn’t have to worry about extensively managing my servers and database. As for deployment, I set up deployment pipelines using github actions and aws code deploy.

Learning all these was easier than I imagined. While use cases differ and we have to choose the best option based on our current situation, using these amazon services has been a good decision for the workflow engine at Dig. We have had no server downtime since deployment.

I made many mistakes before finally following best practices of DevOps. But it has given me many unique insights that one gets only after failing. There are tons of high quality resources online including AWS academy, which I benefited the most from.

Thanks for reading my article, if you have any problem, you can reach out to me in linkedin or in the comments below.

Automation of Images and Orchestration in AWS EKS Part 1

Deepak Poudel — Tue, 28 Mar 2023 16:38:15 +0000

In this blog, we will be discussing five AWS services: AWS EKS (Elastic Kubernetes Service), AWS ECR (Elastic Container Registry), AWS Cloud9 IDE, AWS CodeCommit, and finally AWS CodePipeline. We'll explore the features and benefits of each service, and how they can be used in combination to build and deploy containerized applications in the cloud. Specifically, we'll look at how EKS provides a managed Kubernetes environment, ECR provides a registry for storing and managing Docker images, Cloud9 provides a cloud-based IDE for development, and CodeCommit provides a managed Git repository for version control. By the end of this blog, you'll have a good understanding of how to use these services to build and deploy containerized applications on AWS.

Thanks to Visual Paradigm for the diagraming tool.

Before getting started, you will need an AWS account, a domain name, and a basic understanding of AWS services, including ECR, EKS, CodeCommit, and CodePipeline.

Create a CodeCommit repository

CodeCommit is a fully managed source control service that makes it easy to host private Git repositories. You can use CodeCommit to store your website's source code and manage version control.

To create a CodeCommit repository, follow these steps:

Navigate to the CodeCommit console in the AWS Management Console.
Click "Create repository."
Give your repository a name and click "Create."

After creating the repo, I pushed to the remote that contains a dockerfile. The buildspec.yml also contains steps to build the docker image, tag it and push it to ECR and then invoke updating the EKS.

Create a ECR Repo

ECR is a fully managed Docker container registry that makes it easy to store, manage, and deploy Docker container images. You can use ECR to store your website's Docker container images and manage version control.

To push the image to ECR you need a project and a valid Dockerfile

FROM httpd:2.4
COPY ./index.html /usr/local/apache2/htdocs/
EXPOSE 80

now build the image

docker build -t ecr-address/imagename:tag

then test it locally

docker run -d --name demo-sv -p 9000:80 ecr-address/imagename:tag`

To create an ECR repository, follow these steps:

Navigate to the ECR console in the AWS Management Console.
Click "Create repository."
Give your repository a name and click "Create repository."

after the ECR repo is created, open the cluster and view the push commands
Generally the steps are build tag and push from a Dockerfile that is in the project

aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com

docker tag e9ae3c220b23 aws_account_id.dkr.ecr.us-west-2.amazonaws.com/my-repository:tag

docker push aws_account_id.dkr.ecr.us-west-2.amazonaws.com/my-repository:tag

Create a Cloud9 environment

Cloud9 is an integrated development environment (IDE) that runs in the cloud. It provides a fully featured Linux environment with a web-based IDE and the ability to run commands in a terminal. You can use Cloud9 to write and test your code, as well as interact with your AWS resources. Please feel comfortable to skip cloud9 and use your local shell, I personally prefer using SSH to set up git.

To create a Cloud9 environment, follow these steps:

Navigate to the Cloud9 console in the AWS Management Console.
Click "Create environment."
Give your environment a name and select the settings you want.
Choose an instance type that suits your needs. For this tutorial, we recommend using the t2.micro instance type.
Click "Create environment."

For this demonstration I will be using my local git bash. Make sure kubectl is installed either in your cloud9 or your local shell.

Create a EKS Cluster

Cluster Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:UpdateAutoScalingGroup",
                "ec2:AttachVolume",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CreateRoute",
                "ec2:CreateSecurityGroup",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:DeleteRoute",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteVolume",
                "ec2:DescribeInstances",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVolumes",
                "ec2:DescribeVolumesModifications",
                "ec2:DescribeVpcs",
                "ec2:DescribeDhcpOptions",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeAvailabilityZones",
                "ec2:DetachVolume",
                "ec2:ModifyInstanceAttribute",
                "ec2:ModifyVolume",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:DescribeAccountAttributes",
                "ec2:DescribeAddresses",
                "ec2:DescribeInternetGateways",
                "elasticloadbalancing:AddTags",
                "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
                "elasticloadbalancing:AttachLoadBalancerToSubnets",
                "elasticloadbalancing:ConfigureHealthCheck",
                "elasticloadbalancing:CreateListener",
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:CreateLoadBalancerListeners",
                "elasticloadbalancing:CreateLoadBalancerPolicy",
                "elasticloadbalancing:CreateTargetGroup",
                "elasticloadbalancing:DeleteListener",
                "elasticloadbalancing:DeleteLoadBalancer",
                "elasticloadbalancing:DeleteLoadBalancerListeners",
                "elasticloadbalancing:DeleteTargetGroup",
                "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
                "elasticloadbalancing:DeregisterTargets",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DescribeLoadBalancerPolicies",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DetachLoadBalancerFromSubnets",
                "elasticloadbalancing:ModifyListener",
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "elasticloadbalancing:ModifyTargetGroup",
                "elasticloadbalancing:ModifyTargetGroupAttributes",
                "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                "elasticloadbalancing:RegisterTargets",
                "elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer",
                "elasticloadbalancing:SetLoadBalancerPoliciesOfListener",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
                }
            }
        }
    ]
}

NodeGroup Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SharedSecurityGroupRelatedPermissions",
            "Effect": "Allow",
            "Action": [
                "ec2:RevokeSecurityGroupIngress",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:DescribeInstances",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:DeleteSecurityGroup"
            ],
            "Resource": "*",
            "Condition": {
                "StringLike": {
                    "ec2:ResourceTag/eks": "*"
                }
            }
        },
        {
            "Sid": "EKSCreatedSecurityGroupRelatedPermissions",
            "Effect": "Allow",
            "Action": [
                "ec2:RevokeSecurityGroupIngress",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:DescribeInstances",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:DeleteSecurityGroup"
            ],
            "Resource": "*",
            "Condition": {
                "StringLike": {
                    "ec2:ResourceTag/eks:nodegroup-name": "*"
                }
            }
        },
        {
            "Sid": "LaunchTemplateRelatedPermissions",
            "Effect": "Allow",
            "Action": [
                "ec2:DeleteLaunchTemplate",
                "ec2:CreateLaunchTemplateVersion"
            ],
            "Resource": "*",
            "Condition": {
                "StringLike": {
                    "ec2:ResourceTag/eks:nodegroup-name": "*"
                }
            }
        },
        {
            "Sid": "AutoscalingRelatedPermissions",
            "Effect": "Allow",
            "Action": [
                "autoscaling:UpdateAutoScalingGroup",
                "autoscaling:DeleteAutoScalingGroup",
                "autoscaling:TerminateInstanceInAutoScalingGroup",
                "autoscaling:CompleteLifecycleAction",
                "autoscaling:PutLifecycleHook",
                "autoscaling:PutNotificationConfiguration",
                "autoscaling:EnableMetricsCollection"
            ],
            "Resource": "arn:aws:autoscaling:*:*:*:autoScalingGroupName/eks-*"
        },
        {
            "Sid": "AllowAutoscalingToCreateSLR",
            "Effect": "Allow",
            "Condition": {
                "StringEquals": {
                    "iam:AWSServiceName": "autoscaling.amazonaws.com"
                }
            },
            "Action": "iam:CreateServiceLinkedRole",
            "Resource": "*"
        },
        {
            "Sid": "AllowASGCreationByEKS",
            "Effect": "Allow",
            "Action": [
                "autoscaling:CreateOrUpdateTags",
                "autoscaling:CreateAutoScalingGroup"
            ],
            "Resource": "*",
            "Condition": {
                "ForAnyValue:StringEquals": {
                    "aws:TagKeys": [
                        "eks",
                        "eks:cluster-name",
                        "eks:nodegroup-name"
                    ]
                }
            }
        },
        {
            "Sid": "AllowPassRoleToAutoscaling",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:PassedToService": "autoscaling.amazonaws.com"
                }
            }
        },
        {
            "Sid": "AllowPassRoleToEC2",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "*",
            "Condition": {
                "StringEqualsIfExists": {
                    "iam:PassedToService": [
                        "ec2.amazonaws.com",
                        "ec2.amazonaws.com.cn"
                    ]
                }
            }
        },
        {
            "Sid": "PermissionsToManageResourcesForNodegroups",
            "Effect": "Allow",
            "Action": [
                "iam:GetRole",
                "ec2:CreateLaunchTemplate",
                "ec2:DescribeInstances",
                "iam:GetInstanceProfile",
                "ec2:DescribeLaunchTemplates",
                "autoscaling:DescribeAutoScalingGroups",
                "ec2:CreateSecurityGroup",
                "ec2:DescribeLaunchTemplateVersions",
                "ec2:RunInstances",
                "ec2:DescribeSecurityGroups",
                "ec2:GetConsoleOutput",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSubnets"
            ],
            "Resource": "*"
        },
        {
            "Sid": "PermissionsToCreateAndManageInstanceProfiles",
            "Effect": "Allow",
            "Action": [
                "iam:CreateInstanceProfile",
                "iam:DeleteInstanceProfile",
                "iam:RemoveRoleFromInstanceProfile",
                "iam:AddRoleToInstanceProfile"
            ],
            "Resource": "arn:aws:iam::*:instance-profile/eks-*"
        },
        {
            "Sid": "PermissionsToManageEKSAndKubernetesTags",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags",
                "ec2:DeleteTags"
            ],
            "Resource": "*",
            "Condition": {
                "ForAnyValue:StringLike": {
                    "aws:TagKeys": [
                        "eks",
                        "eks:cluster-name",
                        "eks:nodegroup-name",
                        "kubernetes.io/cluster/*"
                    ]
                }
            }
        }
    ]
}

Use these roles for cluster and the nodegroup. If you are on cloud9 make sure it can access the resources.

now configure AWS cli

aws configure

WARNING! don't use role to create the cluster, you can create the cluster but later can't issue kubectl commands. Use the same user to create the cluster and manage the cluster.

EKS is a fully managed Kubernetes service that makes it easy to deploy, manage, and scale containerized applications. You can use EKS to deploy your website's containerized application and manage its infrastructure.

To create an EKS cluster, follow these steps:

Navigate to the EKS console in the AWS Management Console.
Click "Create cluster."
Choose a Kubernetes version and give your cluster a name.
Choose the VPC and subnet where you want to deploy your cluster.
Choose the security group for your cluster and click "Create."

i have minikube setup and this is the status of my local machine

I have to create a kubeconfig because currently it is configured at my local minikube cluster

I just backed up my old config file and generated a new config inside .kube

aws eks update-kubeconfig --region region-code --name my-cluster

now let's add nodegroup and assign the role to the nodegroup. you can also configure auto scaling on the nodegroup. i've used spot instances

now you can see two instances as the nodes and no pods in the default namespace

let's run image from ECR

either deploy from a yaml or let the kubectl generate a yaml for you. Here's the yaml that was generated for me previously.

apiVersion: v1
items:
- apiVersion: apps/v1
 kind: Deployment
 metadata:
   annotations:
     deployment.kubernetes.io/revision: "1"
   creationTimestamp: "2023-02-02T05:55:05Z"
   generation: 4
   labels:
     app: my-eks
   name: my-eks
   namespace: default
   resourceVersion: "41352"
   uid: 90024fca-9212-40c5-9546-d9b85dfdd98f
 spec:
   progressDeadlineSeconds: 600
   replicas: 2
   revisionHistoryLimit: 10
   selector:
     matchLabels:
       app: my-eks
   strategy:
     rollingUpdate:
       maxSurge: 25%
       maxUnavailable: 25%
     type: RollingUpdate
   template:
     metadata:
       creationTimestamp: null
       labels:
         app: my-eks
     spec:
       containers:
       - image: ecr-address/imagename:tag
         imagePullPolicy: Always
         name: my-eks
         ports:
         - containerPort: 80
           protocol: TCP
         resources: {}
         terminationMessagePath: /dev/termination-log
         terminationMessagePolicy: File
       dnsPolicy: ClusterFirst
       priorityClassName: system-node-critical
       restartPolicy: Always
       schedulerName: default-scheduler
       securityContext: {}
       terminationGracePeriodSeconds: 30
 status:
   availableReplicas: 2
   conditions:
   - lastTransitionTime: "2023-02-02T05:55:05Z"
     lastUpdateTime: "2023-02-02T05:55:45Z"
     message: ReplicaSet "my-eks-c86b768" has successfully progressed.
     reason: NewReplicaSetAvailable
     status: "True"
     type: Progressing
   - lastTransitionTime: "2023-02-02T08:39:56Z"
     lastUpdateTime: "2023-02-02T08:39:56Z"
     message: Deployment has minimum availability.
     reason: MinimumReplicasAvailable
     status: "True"
     type: Available
   observedGeneration: 4
   readyReplicas: 2
   replicas: 2
   updatedReplicas: 2
kind: List
metadata:
 resourceVersion: ""

Ready to create a deployment ?
let's do it from the yaml

kubectl create -f my_deployment_file.yaml

kubectl create deployment my-eks --image=ecr-address/image:tag

now expose the deployment

kubectl expose deployment my-eks --type=LoadBalancer --port=80 --target-port=80

type=loadbalancer we can leverage the cloud provider's load balancer. it's elastic load balancer in case of AWS.

now it's just one replication, let's scale the application to run four replicas.

kubectl scale --replicas=4 deployment my-eks

the URL exposes deployment. Your image will be load balanced and is accessible with the external public facing URL.

kubectl get services

Finally Create a Pipeline

Create a CodePipeline pipeline

CodePipeline is a fully managed continuous delivery service that makes it easy to automate your release process. You can use CodePipeline to create a pipeline that automatically builds, tests, and deploys your website.

I feel like splitting the content in two blogs because it's been long. Automation will be in the next one. Stay tuned. The next blog will continue from this point.

Serverless API Deployment with AWS Lambda and API Gateway

Deepak Poudel — Sun, 26 Feb 2023 16:24:45 +0000

Serverless API Deployment with AWS Lambda and API Gateway
Introduction:
Serverless API deployment with AWS Lambda and API Gateway is a cloud computing model in which an application runs on a serverless architecture without the need to manage servers, operating systems, or infrastructure. With AWS Lambda, developers can write code in response to various triggers, such as an HTTP request, and the code will execute without requiring a server to be running at all times. AWS API Gateway is a fully managed service that enables developers to create, publish, and manage APIs at scale.

In a serverless API deployment, AWS Lambda functions act as the backend code for an API Gateway API. The API Gateway is responsible for handling requests from clients and forwarding them to the appropriate AWS Lambda function. The Lambda function performs the necessary operations and returns a response to the API Gateway, which sends the response back to the client.

The benefits of serverless API deployment with AWS Lambda and API Gateway include:
● Reduced operational overhead: With a serverless deployment model, there is no need to manage servers, operating systems, or infrastructure, which reduces operational overhead.
● Scalability: Serverless architectures can scale automatically in response to changes in demand, allowing APIs to handle many requests without needing manual scaling.
● Cost-effectiveness: Serverless APIs can be cost-effective because they are charged based on the number of requests and the execution time rather than the amount of server time used.
● Ease of development: AWS Lambda and API Gateway provide a simple and intuitive interface for developing, testing, and deploying serverless APIs, which can speed up the development process.
● Security: AWS Lambda and API Gateway provide robust security features, such as identity and access management, encryption, and monitoring, to help protect serverless APIs from security threats.
Pricing:
AWS Lambda:
AWS Lambda pricing is based on the number of requests and the duration of each function execution. AWS Lambda offers a generous free tier of 1 million requests and 400,000 GB-seconds of monthly compute time. After that, pricing is based on the number of requests, the duration of each execution, and the amount of memory the function uses.

API Gateway:
API Gateway pricing is based on the number of API calls and data transfers out. There is a free tier of 1 million API calls per month and up to 12 months of free data transfer out. After that, pricing is based on the number of API calls and the amount of data transferred out.

Steps:

1.Choose to Create a function

1 Create a function
● Choose an Author from scratch
● choose runtime: AWS Lambda supports various programming languages, including Java, Python, Node.js, Go, and others. You can choose the language that best fits your needs and expertise.
● Choose Architecture: x86_64
Expand Change default execution role and check to Create a new role with basic Lambda permission

● You will see a page similar to the page below

Modify the default lambda_function ● We will modify the lambda function to call API using the requests library ● I am taking a dummy API from https://dummyjson.com/

● The code looks like this

Code :

import json
import requests
def lambda_handler(event, context):
base_url = 'https://dummyjson.com/'
product_id = '1'
resp = requests.get(url=base_url+'products/'+product_id)
if resp.status_code != 200:
raise Exception('Failed to retrieve product: '+resp.text)
return {
'statusCode': resp.status_code,
'body': resp.json()
}
Q. what this code does?
1.Imports the necessary modules (JSON and requests).
2.Defines a base URL and a product ID that will be used to construct the API URL.
3.Sends a GET request to the API using the requests library, passing in the full API URL.
4.Checks the response status code to make sure that the request was successful (status code 200).
5.If the response status code is not 200, raise an exception with an error message that includes the response text.
6.If the response status code is 200, return a dictionary containing the response status code and the parsed JSON response body.
●Test the function.

Create a test event

● You will get an error as shown:

Q. why do we get this error, and how to solve it?
➢ This is because there are no library-named requests. Now here comes the use of lambda layers.

Lambda layers:
AWS Lambda Layers are a way to manage common code and libraries in AWS Lambda functions. A layer is essentially a ZIP archive that contains libraries, custom runtimes, or other dependencies that multiple functions in your AWS account can use. Instead of bundling all the code and dependencies within each Lambda function, you can create a layer that includes common libraries or code and then associates that layer with one or more functions. This can help reduce the size of your function code and simplify updates to shared code or libraries.

Fix the error ● Goto your local system and create a directory ● Install the required libraries in our case requests library ● Zip the folder containing libraries

● Goto lambda dashboard, Select layers, and click create layer

● Give name, description and upload the file you zipped in the above step
● Choose compactable architecture and runtimes and create

Add layer

1.Test the lambda function

Now the code is successfully executed.
1.Create an API Gateway
● Goto AWS Management Console and navigate the API Gateway service

Choose REST API

● Create API

●Goto actions and Create Method

●Select GET , Setup and Save
-We are integrating with Lambda function so choose integration type to lambda function
-Choose Lamba region
-Choose the Lamba created in above
-Save

● Add Permission to Lambda Function

Test

You can see the response of our API is according to our desire

●Enable CORS:
Enabling CORS (Cross-Origin Resource Sharing) in API Gateway allows web applications from different domains to access resources served by your API.

Without CORS, web applications can only make requests to endpoints on the same domain where the application is hosted. If you enable CORS, you allow requests to be made from other domains, which can be useful for building web applications that use APIs hosted on a different domain.

● Deploy API
-Give stage name and description(optional)

We have successfully deployed our API and got the invoke URL
● Execute Lambda function when API is triggered
-Goto Lambda page and click Add trigger

-Set trigger configuration as shown below

● Paste the invocation URL in the browser you will see the desired output in JSON format

Conclusion:
In conclusion, AWS Lambda and API Gateway are powerful services that can be used to build and deploy scalable, cost-effective APIs quickly and easily. With Lambda, you can run your code without provisioning or managing servers, and only pay for the computing time you consume. API Gateway makes it easy to create and manage APIs that can be accessed by clients over the internet, with features such as authentication, rate limiting, and caching built in.

Docker Multi Container GitHub Action Deployment Pipeline using AWS Elastic Beanstalk

Deepak Poudel — Tue, 30 Aug 2022 17:54:05 +0000

In this article, we want to build multiple containers using GitHub actions and deploy them to Amazon Elastic Beanstalk. Containers are the new de facto method used in the industry to avoid the most used complaint during the QA phase which is “It Works on my Machine”. It needs some engine such as docker to run. We can describe our container on a simple file that has information about the operating system to use, runtimes, and the artifacts that need to execute on the virtual machine.

Let’s get started. You can find the code in the GitHub link below
GitHub Link
After you have downloaded you can see the files below

Clone the project or download as zip. to run the project locally, you can use docker-compose-dev.yml file with the command in the project’s directory.
This project has a separate docker-compose-dev.yml file because our docker-compose.yml has production version which downloads container from a repository (Docker hub for now). By default, the repository used in docker hub. The repository could be hosted in AWS Elastic Container Repository as well. To do so all we need to do is generate docker config file by logging in and generating AWS credentials file and changing the tag from docker username to awsurl/imagename. And update the repository url in the production docker-compose.yml.
docker-compose -f docker-compose-dev.yml up

It then starts downloading the dependencies and running the command in Dockerfile such as copy, add npm install etc. in a series of containers one after another and build images of those containers.
you can see the containers and images with the command not necessarily in the project’s directory i.e anywhere
We now have a project that has redis and postgres specified as external container dependencies and a react client, nodejs server and an nginx that exposes http port 80.

docker ps –all

docker images

The architecture

Networking between the containers
Networking is a pretty vast topic to discuss. For now, we don’t have to worry about inter container networking at this level because all containers are a part of default bridge network driver. Because of the bridge, and specified hostname, containers can pinpoint other containers and communicate over a virtual network. We have leveraged the use of hostnames in our dockerfile.

The Solution
The solution offered by the project is calculate Fibonacci value for nth position. It uses react for frontend website, redis for cache so that n-1 values can be retrieved if already calculated. Calculated values are stored in a postgres database. Nginx container to serve load balanced http traffic.

Client
The client is a react project that queries the API for previously calculated values and requests the worker container to evaluate Fibonacci value via API. The Container is built in two stages. The first stage is called builder which uses node:16-alpine to install dependencies and build the react bundle. After the bundle is build, HTML, CSS and JavaScript chunks are copied to another container using nginx as image which exposes port 80 internally

Dockerfile.dev is also included so that we could stop pushing to Elastic Beanstalk if the tests fail.

The client nginx listens on port 3000. Here’s a catch; “container Nginx and Nginx inside the client are different entities”. Container nginx listens on port 80 whereas nginx inside the client is a secondary client that is served by container nginx:80. Confusing? I bet it is. Repeat reading the statement.

Nginx
Nginx is a container that serves as the server handler to send and receive http traffic to the services. It exposes port 80. Two Upstreams api with port 5000 and client with port 3000 are attached to the listener at port 80

Server
The server is a container that communicates with postgres and redis via environment provided in the docker-compose.yml file. It contains endoints to fetch values that are calculated and post values to be calculated. It also has a caching logic.

It creates a subscription on the redis client for the given index and then add the index to the postgres. So instead of making query to the postgres, which is definitely more resource intensive than querying from redis, the server can fetch value via the redis used as a cache.
Worker
The worker is a nodejs application that recursively calculates Fibonacci values when a message is received in a subscription topic and publishes it to the redis cache.

Github Action

Login to docker and generate an access token and copy it in the clipboard so that we can use it as docker password in the github actions secret.
Docker username, password, AWS access and secret keys should be added to action secrets in the github repository.
The github workflow is triggered when push is made on the main branch as defined in the workflow file. It logs in to the docker hub, builds the container from the artifact and pushes it to the docker repository.

Create and Elastic Beanstalk Application and add the application name, environment name and the region to github workflow file. Create an IAM that can access the bucket and Elastic Beanstalk with programmatic access and also add it to the github workflow file. Check s3 for bucket created by Elastic Beanstalk and add the bucket name to the github workflow file.
The docker engine expects docker-compose.yml file to configure which containers to run. Since our docker file contains the list of services (i.e. containers). Our application should be live. Try accessing URL which is automatically generated by elastic Beanstalk.

awscommunity #awscommunitybuilders #aws #awssolutionarchitect #awscloud #awscertified #cloudcomputing #awscertification #awstraining #amazonwebservices #cloud #container #eks #ecs #ebs