DEV Community: Asna Siji

Solving it the AWS way.

Asna Siji — Mon, 03 Oct 2022 07:31:30 +0000

My kids love to write and draw. To reduce the screen time and to improve their writing skills, I encourage them to scribble their thoughts on paper. So far so good. But keeping the hard copies is a bit difficult. I would like to convert these handwritten notes to soft copies in textual format [Foreseeing some Kindle plans, when it becomes a small little bundle :-) ].

Let’s solve this the AWS way.

Solution Description

Let’s build a solution leveraging S3 replication / event notification, a serverless compute layer with lambda function and Textract. The sample architecture looks like this.

Two buckets are created, source bucket and destination bucket. Source bucket will hold the images user is giving as input.
The drawings are uploaded to the source bucket with tags say ‘picture’. S3 same region replication is enabled on the source bucket on the configured tag.
A folder is created say ‘story’ in source bucket to upload the handwritten text. S3 event notification is enabled on the source bucket on prefix ‘/story’, in such a way that any upload to this folder will create an event notification.
A lambda function is set up to receive these event notifications. The lambda function will invoke Textract passing the uploaded image as input. Textract will scan the image and return the identified words in the document to the lambda function.
Lambda function logic will consolidate it into sentences and save it to destination bucket as text.
Event notification is enabled on the destination bucket to invoke SNS to send external emails once the process is completed.

Serverless Features in Spot Light
S3 Replication

Amazon Simple Storage Service (S3) can automatically replicate S3 objects to help you reduce costs and protect data.
Two main replication options are:
Cross-Region Replication (CRR) — copies S3 objects across multiple AWS Regions.
Same-Region Replication (SRR) — copies S3 objects between buckets in different availability zones (AZs), in the same region.
S3 also offers a Replication Time Control (RTC) that guarantees object replication in less than fifteen minutes.

S3 Event Notification

Amazon S3 Event Notifications feature helps you to receive notifications when certain events happen in your S3 bucket.
To enable this, a notification configuration to be added with events details that you want S3 to publish and a destination where you want S3 to send the notifications.

Computation Layer with Lambda

AWS Lambda is the serverless event-driven computation layer that runs your code without provisioning or managing servers

Textract

Amazon Textract is a service that can be used to automatically extracts text, handwriting, and data from documents /images.
It uses advanced machine learning (ML) algorithms to achieve this.

SNS

Amazon Simple Notification Service (Amazon SNS) is a fully managed messaging service. It enables communication between applications with a pub/sub functionality. It also supports application to person communications by sending messages to users at scale via SMS, mobile push, and email.

Let’s Build.
Step 1 : Create Buckets

Navigate to the S3 console and create 2 buckets in the same region, say ‘textractsourcebucket’ and ‘textractdestinationbucket’ with versioning enabled.

Step 2 : Creating Roles

We need 2 roles, one for S3 replication and another for lambda function.

Role 1 : Let the S3 replication feature itself create a new role for this. Skip this for now.

Role 2 : Create a role with Cloud Watch, Textract and S3 access for lambda.

Step 3 : Enabling replication

Navigate to the Management tab of the source bucket and create replication rules.

Create replication rule by providing a name. Enable versioning if you have not enabled it before.

We are filtering the replication items based on tag. The drawings will be uploaded with a tag ‘picture’.

Select the destination bucket created before in Step 1 as destination.

Allow S3 replication feature to create a role for you with required permissions.

Save the replication configuration.

You can choose not to replicate existing objects.

Step 4 : Creating lambda function

The lambda function will receive the event notification from S3 when handwritten note is uploaded. It will call Textract to extract the textual details and consolidate it and save to destination bucket as text file.

Navigate to lambda console and create a new function with run time as Node.js.

Change default execution role as the existing role and select the role created in Step 2 Role 2 for lambda. Copy the contents of index.js from below GitHub repo (after making modification, like region, bucket name etc. if any) to the code section of lambda.

https://github.com/asnakhader/textract

Create a test event to test the function. Sample given below (obtained from Cloud Watch logs when the S3 event notification happens).

{
    "Records": [
        {
            "eventVersion": "2.1",
            "eventSource": "aws:s3",
            "awsRegion": "ap-south-1",
            "eventTime": "2022-09-23T04:59:35.441Z",
            "eventName": "ObjectCreated:Put",
            "userIdentity": {
                "principalId": "AWS:BBBBBBB"
            },
            "requestParameters": {
                "sourceIPAddress": "00.00.00.174"
            },
            "responseElements": {
                "x-amz-request-id": "XDG0P50ZJ02XX7JR",
                "x-amz-id-2": "c+xP2nQ780GjtppxeXERgXK9OJt7ZTLUqR941EE9/y74GhIBoQX5ZRb2leJWD40XpkFFK82HZR/lfT/lTR0n/Atnr26OmFK5wzx3F7npKJc="
            },
            "s3": {
                "s3SchemaVersion": "1.0",
                "configurationId": "storyUpload",
                "bucket": {
                    "name": "<your source bucket name>",
                    "ownerIdentity": {
                        "principalId": "BBBBBBB"
                    },
                    "arn": "arn:aws:s3:::<your source bucket name>"
                },
                "object": {
                    "key": "<your object name>",
                    "size": 1522574,
                    "eTag": "c5fa617910f9118efa01ddcbd82e1433",
                    "versionId": "Q9yd0M4L2otyIdURpFOgnZdi46C54y91",
                    "sequencer": "00632D3D375CC84842"
                }
            }
        }
    ]
}

Test the function to see if return a success response.

Step 5: Setting up S3 Event Notification

Navigate to the S3 console and select the source bucket.

Create a folder, say ‘images; to upload handwritten notes.

Navigate to the ‘Properties’ tab and create event notifications.

Create event notification with prefix as /images and suffix as .jpg as we the handwritten notes will be uploaded to this folder as .jpg image.

Enable this for all object creation events and set the destination as lamda function created before.

Step 6 : Create SNS Topic

Navigate to the SNS Console and create a topic.

Navigate to the Access Policy tab and attach a policy which allows the S3 bucket to publish messages to SNS.

Sample policy is given below.

gist.githubusercontent.com

Create a subscription for this topic with an email end point.

Confirm the subscription by accepting the invitation received in email.

Step 7 : Setting up email notification for process completion

Navigate to the destination bucket and create event notification.

Select destination as the previously created SNS topic and save changes.

Step 8 : Testing

Navigate to S3 Source Bucket and upload a drawing with tag as ‘picture’.

Source Bucket
Upload the handwritten text images in folder ‘/images’.

Navigate to destination bucket and check the details. S3 replication has replicated the uploaded image with tag ‘picture’.

Check the details inside the image folder.

The lambda function with Textract has converted the handwritten notes uploaded as .jpg images into textual format. An email notification is received indicating process completion. Woohoo!

Final Product

Excuse the spelling / grammar mistakes pls. Big shoutout to Textract for reading this :-)

At the end of the day, the whole purpose of technology is to ease human lives and save it from monotonous and repetitive work. Do you agree?

Tail End: Now that the kids have taken the pictures of all their stories and given to me, the ball is in my court to convert it and keep it safe.

Auto scaling with custom metrics.

Asna Siji — Sat, 10 Sep 2022 05:17:56 +0000

One of the primary shortcomings of the traditional on premise infrastructure is its inability to scale on demand. Many data centers have unused capacity in store anticipating spikes in load / usage.

Auto Scaling or the ability to automatically adjust capacity to maintain predictable performance thus becomes a primary advantage of using a cloud infrastructure.

AWS Auto Scaling enables to setup application scaling for multiple resources across multiple services including virtual servers (EC2),container tasks(ECS) or NoSQL database like Amazon Dynamo DB. Amazon EC2 Auto Scaling helps to automatically launch or terminate EC2 instances as per the conditions defined.

The most common use case in EC2 Auto Scaling is to configure CloudWatch alarms to launch new EC2 instances when a specific metric exceeds a threshold. This can be done with pre-defined metrics like CPU Utilization, Network In/Out or Customs metrics like memory usage.

Let’s setup EC2 Auto Scaling with a custom metric.

Step 1 : Launch an instance
First, launch an Amazon EC2 Linux instance with default configuration.

Step 2 : Create a role with CloudWatch access
Navigate to IAM and create a role which can be assumed by the EC2 instance which enables Cloud Watch access for EC2. Select the Trusted entity type as AWS service and Use case as EC2.

For the purpose of this demo, I am assigning full access. But always remember ‘PLP’ , Principle of Least Privilege.

Navigate to EC2 instance dashboard and assign this role to the instance by selecting the instance and modifying IAM role under Security option.

Modify the role by selecting the role created in previous step.

Step 3: Install agents
Connect to the instance with EC2 Instance connect.

Its time to leverage some of your Linux admin skills :)

Install Cloud Watch agent to send the EC2 instance metrics to CW for monitoring. The CloudWatch agent is available as a package in Amazon Linux .
sudo yum install amazon-cloudwatch-agent

2.It’s important that EPEL repo is enabled in Linux 2 AMI by running
sudo amazon-linux-extras install epel -y

3.Let’s create some stress.Install the stress package by

sudo yum install stress -y

Create the agent configuration file

The agent configuration file is a JSON file that specifies the metrics and logs that the agent is to collect, including custom metrics. We can create it by using the wizard or manually from scratch. We are creating a sample file as it gives more control over the metrics collected and can specify metrics not available through the wizard.

Commands to create an agent config file named amazon-cloudwatch-agent.json



--run as root user
sudo su

--create the agent config file
touch amazon-cloudwatch-agent.json

--change the file permissions to make it editable
chmod +rwx amazon-cloudwatch-agent.json

--edit the file in visual editor
vi amazon-cloudwatch-agent.json

--Copy the sample contents for amazon-cloudwatch-agent.json and enter :wq command to save the file and exit

--display the file content
cat amazon-cloudwatch-agent.json

Sample for amazon-cloudwatch-agent.json

{ "metrics": { "append_dimensions": { "InstanceId": "${aws:InstanceId}" }, "metrics_collected": { "mem": { "measurement": [ "mem_used_percent" ] }, "disk": { "resources": [ "*" ], "measurement": [ "used_percent" ] } } } }

Any time you change the agent configuration file, you must then restart the agent to have the changes take effect. After you have created a configuration file, you can save it manually as a JSON file and then use this file with the agent on your servers.

5. Start the CloudWatch agent on the server.

Enter following command. Replace configuration-file-path with the path to the agent configuration file.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:configuration-file-path

In this command, -a fetch-config causes the agent to load the latest version of the CloudWatch agent configuration file, and -s starts the agent. Example :

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:amazon-cloudwatch-agent.json

This file is the amazon-cloudwatch-agent.json if you created it manually as per above step 4.

Step 4: Stop the server and create an AMI
Note: AMI means Amazon Machine Image which is a customization of an EC2 instance. We can add own software, configuration, operating system etc. AMI is built for a specific region. This helps for faster boot / configuration time because all your software is pre-packaged.
EC2 instances can be launched from
• A Public AMI: AWS provided
• Your own AMI: you make and maintain them yourself (like what we are doing now)
• An AWS Marketplace AMI: an AMI someone else made

During AMI creation time, a snapshot of the volume will be taken. There is provision to tag the image and snapshot together or separately.

Navigate to the AMI section and wait for the newly created AMI to be ‘Available’.

Step 5: Create a Launch Config
Navigate to EC2 → Launch configurations and create a new launch configuration specifying the previously created AMI and instance type as t2.micro. Assign the role which has Cloud Watch access for Instance profile. .

Assign any existing key pairs and security groups with SSH access and create launch configuration.

Step 6: Auto Scaling Time
Note:An Auto Scaling Group (ASG) helps to maintain a steady performance for the deployed applications. Some of its features are

•Scale out (Launch EC2 instances) to match an increased load
• Scale in (Terminate EC2 instances) to match a decreased load
• Ensure we have a minimum and a maximum number of EC2 instances running
• Automatically register new instances to a load balancer.
• Re-create an EC2 instance in case a previous one is terminated (say unhealthy)

Create an auto scaling group with the Launch Config created in previous step.

Keep the VPC setting as default and choose the subnets where these instance need to be launched. We are opting for a no load balancer option with health checks on EC2 at a grace period of 300 s.

Choose the desired / minimum and maximum capacity needed.

Desired capacity: Represents the initial capacity of the Auto Scaling group at the time of creation. An Auto Scaling group attempts to maintain the desired capacity.
Minimum capacity: Represents the minimum group size.An Auto Scaling group cannot decrease its desired capacity lower than the minimum size limit.
Maximum capacity: Represents the maximum group size.An Auto Scaling group cannot increase its desired capacity more than the mazximum size limit.

Step 7: ASG is launching an instance as per the config

Step 8: Monitor it
Navigate to the CW dashboard.

Under the Metrics → All metrics, we can see that memory utilization is getting collected.

Step 9: Create an Alarm

Create an alarm based on the memory used percent custom metric. For Notifications, choose the default SNS topic for Cloud Watch Alarms and enter your email id to get the notification in case of threshold breach.

Step 10: Link ASG with Alarm
Navigate to the EC2 console and select the ASG from the Auto scaling groups and create a dynamic scaling policy for it.

Choose Step scaling and link the previously created Cloud Watch alarm.Define the action to take. Example :- Add 2 EC2 instances when memory percentage is greater than 50%. Sample given below.

Step 11: Stressing it out
Connect to the live instance with EC2 Instance Connect and run below command to generate the stress

sudo stress — cpu 8 — vm-bytes $(awk ‘/MemAvailable/{printf “%d\n”, $2 * 0.9;}’ < /proc/meminfo)k — vm-keep -m 1

Step 12: Wait for the Alarm
The load is slowly increasing and the graphed metrics for memory usage is indicating a spike.

The alarm state is changing from OK to Alarm.

An email message is received indicating an overflow.

Navigate to the EC2 dashboard and check for new instances.
Yes. the number of instances is increased to 3. ASG has launched 2 additional instances as per the auto scaling config because of the CW alarm.

Now stop the stress and observe the Cloud Watch Alarm. The status is changed from Alarm to OK.

Step 13: Clean up!
Now to clean up, if you stop the instances manually, ASG will again spin up new instances as per the desired capacity. Instead set the minimum and desired capacity in ASG as zero.

This will cause the instances to be automatically terminated by ASG.

Cool. That’s it. We have successfully completed EC2 Auto scaling with Cloud Watch using a custom metrics.