DEV Community: Anudeep Rallapalli

Monitoring Linux EC2 Instance using Prometheus and Grafana

Anudeep Rallapalli — Sat, 10 Apr 2021 19:27:59 +0000

Grafana has become the world’s most popular technology used to compose observability dashboards with everything from Prometheus & Graphite metrics, to logs and application data to power plants and beehives.
• Customers use its open-source analytics and interactive visualization web application.
• It provides charts, graphs, and alerts for the web when connected to supported data sources.
Today, let’s look at building a Grafana dashboard for an Amazon Linux EC2 Instance using Prometheus as Data Source.

Prometheus uses Node_exporter to scrap data from EC2 instance and display them in Grafana dashboards.
• Let’s launch an EC2 Linux Instance from the AWS Console. Make sure, that the security group is allowing all the traffic into the instance (Generally, opening the instance to all the traffic is not advisable for Production based systems).
• SSH into the EC2 instance.

• Now, let's install node_exporter onto the EC2 Instance.
• Run the following command to change the directory to /opt/
cd /opt/
• Now, run the following command to download the node_exporter binary file:
sudo wget https://github.com/prometheus/node_ex porter/releases/download/v1.1.2/node_exporter-1.1.2.linux-amd64.tar.gz
• Run the following command to extract the downloaded file:
sudo tar xf node_exporter-1.1.2.linux-amd64.tar.gz
• Rename the file name “node_exporter-1.1.2.linux-amd64” to “node-exporter” for easy execution.
Run the following command change the name:
sudo mv node_exporter-1.1.2.linux-amd64 node_ex porter
• Let’s start the node-exporter agent on the server by running the following command:
cd node_exporter
sudo ./node_exporter
• Go back to the EC2 Management Console and copy the Public IP address of the EC2 Instance.
• Run the IP Address on your browser with the extension :9100

If you click on "Metrics" button. You would see all the metrics displayed on the screen.

Now, we need to launch two instances from the console. One for Prometheus and the other for Grafana.

• SSH into the Prometheus Instance.
• We need to install the Prometheus agent into the EC2 Instance. Run the following command to install Prometheus.
cd /opt/
sudo wget https://github.com/prometheus/prometheus/releases/download/v2.26.0/prometheus-2.26.0.linux-amd64.tar.gz
• Let’s extract the downloaded file by running the following command:
sudo tar xf prometheus-2.26.0.linux-amd64.tar.gz
• We will be renaming the file “prometheus-2.26.0.linux-amd64” to “prometheus” for easy access.
sudo mv prometheus-2.26.0.linux-amd64 prometheus
• Change the directory to prometheus:
cd prometheus/
• Running the below command will open the Prometheus.yml file in a text editor.
sudo vi prometheus.yml

• In the text editor, enter the value of the EC2 instance’s IP address at “target” under “static_configure” with extension :9100 and save the file.

• Enter the Public IP address of the Prometheus server with the extension :9090 in your browser.

• You should be able to see the Prometheus Dashboard page.

• Select “Targets” in dashboard.

Its time to configure our final instance for Grafana.
• SSH into the Grafana instance which we already created.
• Run the below command in the SSH prompt to install Grafana binaries into the server. It will open a text editor
sudo vi /etc/yum.repos.d/grafana.repo
• Paste the below text in the editor to provide the configuration for the Grafana server. Save and close the text editor
[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
• Time to install, start and enable Grafana. Run the below commands one after the other:

sudo yum install Grafana
sudo systemctl start grafana-server
sudo systemctl enable grafana-server
• Now, type the Public IP address of the Grafana instance in the browser with the extension :3000
• Enter ‘admin’ as username and password.

• You will be prompted to change the password. Enter the new password and confirm it.
• Now, select Prometheus Private IP address as data source.

• Since both Grafana and Prometheus are available in the same network. We use private IP address for Prometheus.

• Select the Dashboard Template “Node_exporter”

Finally, the Grafana Dashboard for EC2 linux instance using Prometheus is ready!

Importance of Security and Infrastructure Governance in AWS

Anudeep Rallapalli — Sat, 10 Apr 2021 11:48:25 +0000

Security is one of the biggest concerns in Cloud Computing. Every company whose application is running on either web or the mobile phone is always worried about the security of their infrastructure.
1) The leadership team at a large Online Store is worried that its employee might delete its critical data by mistake.
2) The CEO of a Social Media company is concerned about the security of its website images stored in the Cloud.
3) The DevOps team of a top Online Gaming Company wants to provide a secure and easy way for its users to login to their site and not worry about the leakage of these login credentials.
4) The Site Reliability team at an Online Media company is concerned about the reliability of its server whenever there is a viral news on their website.
5) The leadership team at a large IT firm is concerned that their employees might have more privileges than required.

These are few of many security questions which gives all of us sleepless nights.
It takes more than one Documentation to answer all of the security questions on Cloud.
For now, Let's look at the above 5 scenarios and try to resolve these concerns - One Concern at a Time!

1) The leadership team at a large Online Store is worried that its employee might delete its critical data by mistake.
Loosing crucial data due to human error is avoidable. We need to automate the infrastructure governance. In AWS, we have Identity and Access Management (IAM).

• It is a framework of policies and technologies for ensuring that the proper people in an enterprise have the appropriate access to technology resources.
• It helps the root user to securely have control access to AWS resources.
• You use IAM to control who is authenticated (signed in) and authorized (has permissions) to use resources.
• To learn more about IAM, click on the following link: https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html

2) The CEO of a Social Media company is concerned about the security of its website images stored in the Cloud.
In AWS, S3(Simple Storage Service) and EBS (Elastic Block Storage) provide storage for front-end and Back-end files of the application. Encryption is possible for the files stored in S3 and EBS. Let’s see how!

S3 (Simple Storage Service):
• S3 is an object-based storage system which stores objects in buckets.
• It provides unlimited object storage where each object can be 0B to 5TB in size.

Security and Encryption in S3:
• By default, all newly created buckets are PRIVATE.
• Access Control can be set by:

Bucket Policy – Bucket Level Encryption.
Access Control Lists – Object Level Encryption.

• S3 buckets can be configured to create access logs.
• It logs all requests made to S3 buckets.
• These logs can be sent to another bucket or even to a bucket on another account.

Encryption in Transit:
• Traffic flows in S3 following HTTPS Protocol.
• Any traffic flowing through HTTPS will be encrypted by SSL/TLS.

Encryption at Rest:
Encryption for data at rest can be done using 3 different keys.
• S3 Managed Keys – SSE-S3
• AWS KMS Managed Keys – SSE-KMS
• Server-Side Encryption with customer provided keys – SSE-C
• Client-Side Encryption – Files are encrypted at the client’s end before sent to S3

For more information about S3 and its Security. Click on the following link: https://aws.amazon.com/s3/security/#:~:text=Amazon%20S3%20offers%20flexible%20security,Private%20Cloud%20(Amazon%20VPC).

EBS-Elastic Block Store:
• It provides persistent block storage volume for use with Amazon EC2 instance in AWS Cloud.
• Each EBS volume is automatically replicated within its AZ to protect you from component failure, offering high availability and Durability.
• Snapshots can be taken for each of the EBS store volumes. They exist on S3.
• Snapshots are point in time copies of volumes. Think of snapshot as photograph of disks.
• Snapshots are incremental – this means that only the blocks that are changed since last snapshot are moved to S3.
• AMIs can also be created from snapshots.

Encryption of EBS Volumes:
Encryption of EBS volumes can be done in two ways.
• EBS volumes can be encrypted during its creation.
• They can also be encrypted after they are created as well by following the below steps.

Create a snapshot of unencrypted root device.
Create a copy of the snapshot and select the encrypt option.
Create an AMI of the encrypted snapshot.
Use AMI to launch new encrypted instances.

For more information about EBS Encryption. Check the following link: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html

3) The DevOps team of a top Online Gaming Company wants to provide a secure and easy way for its users to login to their site and not worry about the leakage of these login credentials.
Every customer wants their login credentials to be very confidential and discreet. At the same time, they also expect the login process to be simple and smooth as well.
AWS provides Web Identity Federation to meet the above demand:
• It allows users to authenticate with the web identity provider (Google, Facebook, Amazon).
• The user authenticates first with the web ID provider and receives an authentication token, which is exchanged for temporary AWS credentials allowing them to assume the IAM role.

• Cognito is an identity broker which handles interaction between your applications and the web ID provider.

It provides sign up, sign in & guest user access.
Syncs user data for a seamless experience across your devices.

• Cognito is the AWS recommended approach for web ID federation particularly for mobile apps.
For more information about Cognito. Click on the following link: https://docs.aws.amazon.com/cognito/

4) The Site Reliability team at an Online Media company is concerned about the reliability of its server whenever there is a viral news on their website.
Automating the monitoring service for 24/7 is one of the main advantages of having Cloud Infrastructure. AWS provides CloudWatch and CloudTrail for monitoring the infrastructure.

CloudWatch:
• It is a monitoring service which monitors your AWS resources as well as applications running on AWS.
• CloudWatch can monitor:

1) Compute services such as:
a. EC2 Instances
b. Auto Scaling Groups
c. Elastic Load Balancers
d. Route 53 Health Checks

2) Storage & Content Delivery:
a. EBS Volumes
b. Storage Gateway
c. Cloudfront

Cloudwatch monitoring in EC2:
• Cloudwatch monitors the following metrics in EC2:

CPU
Network
Disk
Status Check

• RAM Utilization is a custom metric.
• CloudWatch with EC2 can monitor events every 5 minutes by default.
• You can have 1 minute interval by turning on detailed monitoring.
• You can create CloudWatch alarms which triggers notifications.
• You can retrieve data from terminated EC2 or ELB instances after its termination.
• CloudWatch logs by default are stored indefinitely.
• CloudWatch services are not restricted to just AWS resources. They can be used on premise as well.
• The user just needs to download SSM agent and CloudWatch agent on the server.
To know more about CloudWatch. Follow the link: https://docs.aws.amazon.com/cloudwatch/index.html

CloudTrail:

• It increases visibility into your user and resource activity by recording AWS management console actions and API calls.
• You can identify which user and accounts called AWS, the source IP address from which call was made and when the call occurred.
For more about CloudTrail. Click on the link: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html
CloudWatch is all about performance and CloudTrail is all about auditing.

5) The leadership team at a large IT firm is concerned that their employees might have more privileges than required.
It is very important especially in large companies that its employees always have least privileges towards AWS resources. The leadership team can use AWS Organizations for such instances in AWS cloud infrastructure.

AWS Organizations:
• It centrally manages policies across multiple AWS accounts.
• It controls access to AWS services within these accounts using Service Control Policies (SCP).
• It also automates AWS account creation and management.
• It consolidates billing across multiple AWS Accounts.

Tagging and Resource Group:
• Tagging every AWS service is very important.
• Resource Groups are a way of grouping tags.
• You can use resource groups with AWS systems manager to automate tasks.

AWS Cost Explorer & Cost Allocation Tags:
• Cost Explorer is a tool that enables you to view and analyze your costs and usage.
• Use tags to tag your resources.
• Configure tags for cost centers (i.e., by department, Emp ID etc.)
• Activate cost allocation tags to track your cost by tags.
For more information about AWS Organizations. Click the following link: https://docs.aws.amazon.com/organizations/

This was about Security and few of many AWS services for security and infrastructure governance in AWS.

Build a Modern Web Application

Anudeep Rallapalli — Sat, 03 Apr 2021 04:54:21 +0000

Q) What is a Modern Application?
A) • Modern applications isolate the business logic, optimizes reuse and iteration, it removes the overhead everywhere possible.
• In plain words, Modern apps are built using services that enable you to focus on writing code while automating infrastructure maintenance tasks.

• Let's look at building one such modern web application using AWS services.
Technologies used:
 AWS Cloud9
 Amazon Simple Storage Service (S3)
 AWS Fargate
 AWS CloudFormation
 AWS Identity and Access Management (IAM)
 Amazon Virtual Private Cloud (VPC)
 Amazon Elastic Load Balancing
 Amazon Elastic Container Service (ECS)
 AWS Elastic Container Registry (ECR)
 AWS CodeCommit
 AWS CodePipeline
 AWS CodeDeploy
 AWS CodeBuild
 Amazon DynamoDB
 Amazon Cognito
 Amazon API Gateway
 AWS Kinesis Firehose
 Amazon S3
 AWS Lambda
 AWS Serverless Application Model (AWS SAM)
 AWS SAM Command Line Interface (SAM CLI)

• We will build a sample website called Mythical Mysfits that enables users to adopt a fantasy creature (mysfit) as pet. Here is the working sample of this website at: www.mythicalmysfits.com
• We will be walking through the steps to create an architected web application.
• We will be hosting this web application on a front-end web server and connect it to a backend database.
• We also will be setting up user authentication and will be able to collect and analyze user behavior.
• The site also provides basic functionality such as ability to “like” your favorite mysfit and reserve your chosen mysfit for adoption.
• It also allows you to gather insights about user behavior for future analysis.

The below application architecture diagram provides a structural representation of the services that make up Mythical Mysfits and how these services interact with each other.

Let's begin by hosting the static content (html, js, css, media content, etc.) of our Mythical Mysfit website on Amazon S3 (Simple Storage Service).
S3 is a highly durable, highly available, and inexpensive object storage service that can serve stored objects directly via HTTP. This makes it wonderfully useful for serving static web content directly to web browsers for sites on the Internet.

• Sign into your AWS console and select the region of your choice.
• Navigate to the Cloud9 page and create an environment with default settings.
• We will be completing all the development tasks required within our own browser using the cloud-based IDE, AWS Cloud9.

Use the below git command in the terminal to clone the necessary code to complete this tutorial:
git clone -b python https://github.com/aws-samples/aws-modern-application-workshop.git
Please, note that the above repository is a property of AWS and they own its entire copyrights.
Now, change directory to the newly cloned repository directory:
cd aws-modern-application-workshop

We are all set to create the infrastructure components needed for hosting a static website in Amazon S3 via the AWS CLI.
• Run the below command in the terminal to create the S3 bucket with the name of your choice.
aws s3 mb s3://REPLACE_ME_BUCKET_NAME
• We will be setting some configuration options that enable the bucket to be used for static website hosting.
• Run the below command in your terminal to make your S3 bucket ready for static website hosting:
aws s3 website s3://REPLACE_ME_BUCKET_NAME --index-document index.html
• By default, the public access will be blocked for the S3 bucket for security reasons.
• We need to enable the public access as this bucket is static hosting by changing the Bucket policy.
• Bucket Policy is a JSON file located at ~/environment/aws-modern-application-workshop/module-1/aws-cli/website-bucket-policy.json.
• Open the file and make the necessary changes such as entering your bucket name.
• Now, Execute the following CLI command to add a public bucket policy to your website:
aws s3api put-bucket-policy --bucket REPLACE_ME_BUCKET_NAME --policy file://~/environment/aws-modern-application-workshop/module-1/aws-cli/website-bucket-policy.json
• Now that our new website bucket is configured appropriately, let's add the first iteration of the Mythical Mysfits homepage to the bucket by running the following S3 CLI command
aws s3 cp ~/environment/aws-modern-application-workshop/module-1/web/index.html s3://REPLACE_ME_BUCKET_NAME/index.html
• Our S3 bucket is up and hosting.

• Open your favorite browser and run the below command with your inputs.
http://REPLACE_ME_BUCKET_NAME.s3-website-REPLACE_ME_YOUR_REGION.amazonaws.com

• Congratulations, you have created the basic static Mythical Mysfits Website!

• Our Mythical Mysfits website needs to integrate with an application backend. So, we will create a new microservice hosted using AWS Fargate.
• We need to create the core infrastructure environment that the service will use, including the networking infrastructure in Amazon VPC, and the AWS IAM Roles.
• To accomplish this, we will be using AWS Cloudformation.
• It is available in the location: /module-2/cfn/core.yml
• Run the below command in the Cloud9 terminal (might take upto 10 minutes for stack to be created):
aws cloudformation create-stack --stack-name MythicalMysfitsCoreStack --capabilities CAPABILITY_NAMED_IAM --template-body file://~/environment/aws-modern-application-workshop/module-2/cfn/core.yml
Navigate to the Cloudformation page in the console and wait for the Status to change to "CREATE_COMPLETE"

The output of this stack will be used a lot later. Hence, we will be saving them in a file named cloudformation-core-output.json by running the below command.
aws cloudformation describe-stacks --stack-name MythicalMysfitsCoreStack > ~/environment/cloudformation-core-output.json

Now, we will be creating a Docker container image that contains all of the code and configuration required to run the Mythical Mysfits backend as a microservice API created with Flask.
We will build the Docker container image within Cloud9 and then push it to the Amazon ECR, where it will be available to pull when we create our service using Fargate.

All of the code required to run our service backend is stored within the /module-2/app/ directory of the repository you've cloned into your Cloud9 IDE.
Docker comes already installed on the Cloud9 IDE that you've created, so in order to build the Docker image locally, all we need to do is run the following two commands in the Cloud9 terminal:

cd ~/environment/aws-modern-application-workshop/module-2/app
docker build . -t REPLACE_ME_AWS_ACCOUNT_ID.dkr.ecr.REPLACE_ME_REGION.amazonaws.com/mythicalmysfits/service:latest
You can find the account ID from the below file:
/cloudformation-core-output.json

Once docker downloads all the necessary dependancy packages, copy the image tag from the output. We will be using it in the future.
It will be in the format: 111111111111.dkr.ecr.us-east-1.amazonaws.com/mythicalmysfits/service:latest

Let's test our image locally within Cloud9 to make sure everything is operating as expected.
Run the following command to deploy the container “locally” (which is actually within your Cloud9 IDE inside AWS):
docker run -p 8080:8080 REPLACE_ME_WITH_DOCKER_IMAGE_TAG
Enter the image tag which you have saved earlier.
We will see the below output, if the container is up and running.
* Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
To test our service with a local request, we're going to open up the built-in web browser within the Cloud9 IDE that can be used to preview applications that are running on the IDE instance.
To open the preview web browser, select Preview > Preview Running Application in the Cloud9 menu bar:
Append /mysfits to the end of the URI in the address bar of the preview browser and hit ENTER:
If successful you will see a response from the service that returns the JSON document stored at /aws-modern-application-workshop/module-2/app/service/mysfits-response.json
With a successful test of our service locally, we're ready to create a container image repository in Amazon Elastic Container Registry (Amazon ECR) and push our image into it.
In order to create the registry, run the following command:
aws ecr create-repository --repository-name mythicalmysfits/service
In order to push container images into our new repository, we will need the authentication credentials for our Docker client to the repository.
Run the following command, which will return a login command to retrieve credentials for our Docker client and then automatically execute it:
$(aws ecr get-login --no-include-email)
'Login Succeeded' will be reported if the command is successful.
Using the below command for the Docker to push the image and all the images it depends on to Amazon ECR:
docker push REPLACE_ME_WITH_DOCKER_IMAGE_TAG
Now, Run the following command to see your newly pushed Docker image stored inside the ECR repository:
aws ecr describe-images --repository-name mythicalmysfits/service

Now, we have an image available in ECR that we can deploy to a service hosted on Amazon ECS using AWS Fargate. So that, the website will be publicly available behind a Network Load Balancer.
First, we will create a cluster in the Amazon ECS.
To create a new cluster in ECS, run the following command:
aws ecs create-cluster --cluster-name MythicalMysfits-Cluster
Next, we will create a new log group in AWS CloudWatch Logs for log collection and analysis.
To create the new log group in CloudWatch logs, run the following command:
aws logs create-log-group --log-group-name mythicalmysfits-logs
Since, we have a cluster created and a log group defined for where our container logs will be pushed to, we're ready to register an ECS task definition.
The input to the CLI command will be available in the below JSON File:
~/environment/aws-modern-application-workshop/module-2/aws-cli/task-definition.json
Replace the values from the image tag and from /cloudformation-core-output.json
Once you have replaced the values in task-defintion.json and saved it. Execute the following command:
aws ecs register-task-definition --cli-input-json file://~/environment/aws-modern-application-workshop/module-2/aws-cli/task-definition.json
Rather than directly exposing our service to the Internet, we will provision NLB to sit in front of our service tier.
To provision a new NLB, execute the following CLI command:
aws elbv2 create-load-balancer --name mysfits-nlb --scheme internet-facing --type network --subnets REPLACE_ME_PUBLIC_SUBNET_ONE REPLACE_ME_PUBLIC_SUBNET_TWO > ~/environment/nlb-output.json
Retrieve the subnetIds from /cloudformation-core-output.json
A new file will be created in your IDE called nlb-output.json.
We will be using the outputs in the file in later steps.
We need to create target groups for the NLB. Run the below command:
aws elbv2 create-target-group --name MythicalMysfits-TargetGroup --port 8080 --protocol TCP --target-type ip --vpc-id REPLACE_ME_VPC_ID --health-check-interval-seconds 10 --health-check-path / --health-check-protocol HTTP --healthy-threshold-count 3 --unhealthy-threshold-count 3 > ~/environment/target-group-output.json
You can find the VPC_ID value in /cloudformation-core-output.json
The output will be saved to target-group-output.json in your IDE. We will reference the output of this file in a subsequent step.
Now, we need to create a Load Balancer Listener for the NLB. Run the following command:
aws elbv2 create-listener --default-actions TargetGroupArn=REPLACE_ME_NLB_TARGET_GROUP_ARN,Type=forward --load-balancer-arn REPLACE_ME_NLB_ARN --port 80 --protocol TCP
You can find the TargetGroup and NLB ARN in the files we saved earlier while creating NLB and TargetGroup.
We need to create an IAM role that grants the ECS service itself permissions to make ECS API requests within your account.
To create the role, execute the following command in the terminal:
aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com
If the above returns an error about the role existing already, you can ignore it, as it would indicate the role has automatically been created in your account in the past.
With the NLB created and configured, and the ECS service granted appropriate permissions, we're ready to create the actual ECS service.
Open ~/environment/aws-modern-application-workshop/module-2/aws-cli/service-definition.json in the IDE and replace the indicated values of REPLACE_ME.
Save it, then run the following command to create the service:
aws ecs create-service --cli-input-json file://~/environment/aws-modern-application-workshop/module-2/aws-cli/service-definition.json
Copy the DNS name you saved when creating the NLB and send a request to it using your favorite browser.
The DNS name will be in the below format:
http://mysfits-nlb-123456789-abc123456.elb.us-east-1.amazonaws.com/mysfits
You will receive a response in JSON format.
Next, we need to integrate our website with your new API backend instead of using the hard coded data that we previously uploaded to S3.
Open the below file and update the NLB URL.
/module-2/web/index.html
Run the following command to upload this file to your S3 hosted website
aws s3 cp ~/environment/aws-modern-application-workshop/module-2/web/index.html s3://INSERT-YOUR-BUCKET-NAME/index.html
Make sure, you update the Bucket name in the command before execution.
Open the website using the S3 link. Now, it is retrieving JSON data from your Flask API running within a Docker container deployed to AWS Fargate.

Now that we have a service up and running, we may think of code changes that we make to our Flask service. It would be a bottleneck for your development speed if you had to go through all of the same steps above every time you wanted to deploy a new feature to your service.
That's where we need CI/CD.

Run the following command to create another S3 bucket that will be used to store the temporary artifacts that are created in the middle of our CI/CD pipeline executions:
aws s3 mb s3://REPLACE_ME_CHOOSE_ARTIFACTS_BUCKET_NAME
Unlike our website bucket that allowed access to anyone, only our CI/CD pipeline should have access to this bucket.
Open the following JSON file and make the relevant changes according to your services from /cloudformation-core-output.json.
Once you've modified and saved this file, execute the following command:
aws s3api put-bucket-policy --bucket REPLACE_ME_ARTIFACTS_BUCKET_NAME --policy file://~/environment/aws-modern-application-workshop/module-2/aws-cli/artifacts-bucket-policy.json
We need to create a CodeCommit repository to push and store the code. Run the following command:
aws codecommit create-repository --repository-name MythicalMysfitsService-Repository
With a repository to store our code in, and an S3 bucket that will be used for our CI/CD artifacts, lets add to the CI/CD stack with a way for a service build to occur using AWS CodeBuild Project.
Replace the values in the below file and save it.
~/environment/aws-modern-application-workshop/module-2/aws-cli/code-build-project.json
Now, execute the following command to create a project:
aws codebuild create-project --cli-input-json file://~/environment/aws-modern-application-workshop/module-2/aws-cli/code-build-project.json
We will use CodePipeline to continuously integrate our CodeCommit repository with our CodeBuild project so that builds will automatically occur whenever a code change is pushed to the repository.
Open the following file, make the changes and save it.
~/environment/aws-modern-application-workshop/module-2/aws-cli/code-pipeline.json
Use the following command to create a pipeline in CodePipeline:
aws codepipeline create-pipeline --cli-input-json file://~/environment/aws-modern-application-workshop/module-2/aws-cli/code-pipeline.json
We need to give CodeBuild permission to push container images into ECR.
Locate the below file and make changes:
~/environment/aws-modern-application-workshop/module-2/aws-cli/ecr-policy.json
Save this file and then run the following command to create the policy:
aws ecr set-repository-policy --repository-name mythicalmysfits/service --policy-text file://~/environment/aws-modern-application-workshop/module-2/aws-cli/ecr-policy.json
Now, we will be moving all of the website data into DynamoDB to make the websites future more extensible and scalable.

We will be creating a DynaomDB table with a primary index and two secondary indexes
These indexes will help us to use the filter option which is used to filter the characters in the website based on their profile.

Run the below command:
aws dynamodb create-table --cli-input-json file://~/environment/aws-modern-application-workshop/module-3/aws-cli/dynamodb-table.json
A DynamoDB Table is created. We can view the details of the table from the below command:
aws dynamodb describe-table --table-name MysfitsTable
Following table will retrieve all of the items stored in the table (you'll see that this table is empty):

aws dynamodb scan --table-name MysfitsTable

We will be using the DynamoDB API BatchWriteItem command to batch insert a number of website's items into the table:
aws dynamodb batch-write-item --request-items file://~/environment/aws-modern-application-workshop/module-3/aws-cli/populate-dynamodb.json
Now, if you run the below command. You will see that there are items loaded into the table.
aws dynamodb scan --table-name MysfitsTable

Since, our data is in the table, we need to modify our application code to read from this table instead of returning the static JSON file.
To copy the new files into your CodeCommit repository directory, execute the following command in the terminal:
cp ~/environment/aws-modern-application-workshop/module-3/app/service/* ~/environment/MythicalMysfitsService-Repository/service/
Now, we need to check in these code changes to CodeCommit using the git command line client.
Run the following commands to check in the new code changes and kick of your CI/CD pipeline:
cd ~/environment/MythicalMysfitsService-Repository
git add .
git commit -m "Add new integration to DynamoDB."
git push
Now, in just 5-10 minutes you'll see your code changes make it through your full CI/CD pipeline in CodePipeline and out to your deployed Flask service to AWS Fargate on Amazon ECS.
Feel free to explore the AWS CodePipeline console to see the changes progress through your pipeline.
Finally, we need to publish a new index.html page to our S3 bucket.
You can find the file in the below location. Make the necessary changes and save the file.
~/environment/aws-modern-application-workshop/module-3/web/index.html.
Run the following command to upload the new index.html file.
aws s3 cp --recursive ~/environment/aws-modern-application-workshop/module-3/web/ s3://your_bucket_name_here/
If you revisit your website. We can find the filter option which means that the data is loading from DynamoDB.
Now, we will create a User Pool in AWS Cognito, to enable registration and authentication of website users.
Then, to make sure that only registered users are authorized to make changes in the website, we will deploy a REST API with Amazon API Gateway to sit in front of our NLB.

Execute the following CLI command to create a user pool named MysfitsUserPool
aws cognito-idp create-user-pool --pool-name MysfitsUserPool --auto-verified-attributes email

Copy the user pool unique ID from the response of the above command.
Next, in order to integrate our frontend website with Cognito, we must create a new User Pool Client for this user pool.
Run the following command (replacing the --user-pool-id value with the one you copied above):
aws cognito-idp create-user-pool-client --user-pool-id REPLACE_ME --client-name MysfitsUserPoolClient
Now, let's turn our attention to creating a new RESTful API in front of our existing Flask service.
We need to configure an API Gateway VPC Link that enables API Gateway APIs to directly integrate with backend web services that are privately hosted inside a VPC using the following CLI command:
aws apigateway create-vpc-link --name MysfitsApiVpcLink --target-arns REPLACE_ME_NLB_ARN > ~/environment/api-gateway-link-output.json
It will take about 5-10 minutes for the VPC link to be created.
A file called api-gateway-link-output.json that contains the id for the VPC Link that is being created.
You can copy the id from this file and proceed on to the next step.
The REST API is defined using Swagger. This Swagger definition of the API is located at ~/environment/aws-modern-applicaiton-workshop/module-4/aws-cli/api-swagger.json.
Open the above location and make the necessary changes for the AWS services.
Once the edits have been made, save the file and execute the following AWS CLI command:

aws apigateway import-rest-api --parameters endpointConfigurationTypes=REGIONAL --body file://~/environment/aws-modern-application-workshop/module-4/aws-cli/api-swagger.json --fail-on-warnings
Copy the response this command returns and save the id value for the next step:
Now, our API has been created, but it's yet to be deployed anywhere.
We will call our stage prod. To create a deployment for the prod stage, execute the following CLI command:
aws apigateway create-deployment --rest-api-id REPLACE_ME_WITH_API_ID --stage-name prod
Now, the API is available in the internet. Use the following link:
https://REPLACE_ME_WITH_API_ID.execute-api.REPLACE_ME_WITH_REGION.amazonaws.com/prod/mysfits
Now, we need to include updated Python code for your backend Flask web service, to accommodates the new functionality of the website.
Let's overwrite your existing codebase with these files and push them into the repository:
cd ~/environment/MythicalMysfitsService-Repository/
cp -r ~/environment/aws-modern-application-workshop/module-4/app/* .
git add .
git commit -m "Update service code backend to enable additional website features."
git push
The service updates are being automatically pushed through your CI/CD pipeline.
Open the location /environment/aws-modern-application-workshop/module-4/app/web/index.html which contains the new version of the website's index.html file.
Make the relevant changes in the additional two html files along with index.html and save the files.
Run the below command:
aws s3 cp --recursive ~/environment/aws-modern-application-workshop/module-4/web/ s3://YOUR-S3-BUCKET/
Refresh the Mythical Mysfits website in your browser to see the new functionality in action

We will be creating a way to better understand how users are interacting with the website.

We will create a new and decoupled service for the purpose of receiving user click events from the website. This full stack has been represented using a provided CloudFormation template.
Let's create a new CodeCommit repository where the streaming service code will live:
aws codecommit create-repository --repository-name MythicalMysfitsStreamingService-Repository
Copy the value of 'cloneUrlHttp' from the above output.
Next, let's clone that new and empty repository into our IDE:
cd ~/environment/
git clone {insert the copied cloneValueUrl from above}
We will be moving our working directory into this new repository:
cd ~/environment/MythicalMysfitsStreamingService-Repository/
Let's copy the new application components into this new repository directory:
cp -r ~/environment/aws-modern-application-workshop/module-5/app/streaming/* .
And finally let's copy the CloudFormation template for this module as well.
cp ~/environment/aws-modern-application-workshop/module-5/cfn/* .
Run the below command to install the requests package and its dependencies locally alongside your function code:
pip install requests -t .
Open the streamProcessor.py file and replace the values of the ApiEndpoint with the actual value.
Now, Let's commit our code changes to the new repository so that they're saved in CodeCommit:
git add .
git commit -m "New stream processing service."
git push
Let's create an S3 bucket where AWS SAM CLI will package all of our function code, upload it to S3, and create the deployable CloudFormation template.
aws s3 mb s3://REPLACE_ME_BUCKET_NAME

Now that our bucket is created, we are ready to use the SAM CLI to package and upload our code and transform the CloudFormation template.
sam package --template-file ./real-time-streaming.yml --output-template-file ./transformed-streaming.yml --s3-bucket replace-with-your-bucket-name
Dont forget to replace the above value with the bucket name.
Let's deploy the Stack Using AWS CloudFormation. Run the below command:
aws cloudformation deploy --template-file /home/ec2-user/environment/MythicalMysfitsStreamingService-Repository/cfn/transformed-streaming.yml --stack-name MythicalMysfitsStreamingStack --capabilities CAPABILITY_IAM
Once this stack creation is complete, the full real-time processing microservice will be created.
With the streaming stack up and running, we now need to publish a new version of our website frontend that includes the JavaScript that sends events to our service whenever a profile is clicked by a user.
Below, is the location of the new index.html file:
~/environment/aws-modern-application-workshop/module-5/web/index.html
Make the relevant changes in the above file and perform the following command for the new streaming stack to retrieve the new API Gateway endpoint for your stream processing service:
aws cloudformation describe-stacks --stack-name MythicalMysfitsStreamingStack
Finally, replace the final value within index.html for the streamingApiEndpoint and run the following command.
Refresh your Mythical Mysfits website in the browser and you have a site that records and publishes each time a user clicks on a profile.

To view the records that have been processed, check the destination S3 bucket created as part of your MythicalMysfitsStreamingStack.

Be sure to delete all the resources created in order to ensure that billing for the resources does not continue for longer than you intend.

How to Build a Real-Time Leaderboard with Amazon Aurora Serverless and Amazon ElastiCache?

Anudeep Rallapalli — Tue, 02 Feb 2021 06:14:26 +0000

Every Kid inside us loves to play Games. Especially the one's where we compete with our peers. We all loved to see our name on the top of that Leaderboard! Lets see how that Leaderboard is build using Amazon Aurora Serverless and Amazon ElastiCache.

What is Amazon Aurora Serverless?
• Amazon Aurora is a highly performant, cloud-native relational database offering from AWS that offers both MySQL-compatible and PostgreSQL-compatible editions.
• The Serverless offering of the Aurora database provides on-demand automatic scaling capabilities as well as the Data API, a fast, secure method for accessing your database over HTTP.

What is the Amazon ElastiCache?
• Amazon ElastiCache is a fully managed, in-memory data store service from AWS for use cases that require blazing-fast response times.
• You can use Redis or Memcached with ElastiCache.

Why use both Amazon Aurora Serverless and ElastiCache for a Game Application?
• Amazon Aurora Serverless and Amazon ElastiCache are both commonly used individually in game applications.
• Together, they provide a combination of top-tier speed of an in-memory cache with the reliability and flexibility of a relational database.
• We will be using Amazon ElastiCache for high-volume performance, low-latency leaderboard checks for different games, and Amazon Aurora Serverless to store all historical data and provide redundancy for the leaderboard data

Technologies Used:
• Amazon Aurora Serverless for data storage, including the Data API for HTTP-based database access from your Lambda function.
• AWS Secrets Manager for storing your database credentials when using the Data API.
• Amazon ElastiCache for data storage of global leaderboards, using the Redis engine and Sorted Sets to store your leaderboards.
• Amazon Cognito for user registration and authentication.
• AWS Lambda for compute.
• Amazon API Gateway for HTTP-based access to your Lambda function.

As part of the application, we need a leaderboard system where users can compare their total score against other players as well as view the leaderboard for specific levels in the game. Finally, a user may want to see all of the scores they’ve received in a particular level.

Let's see how we use Amazon Aurora Serverless and Amazon ElastiCache to handle these access patterns.

• Set up an AWS Account and Configure an AWS Cloud9 Instance.

• Provision Amazon Aurora Serverless database and save the database credentials to AWS Secret Manager for use with Data API.

• Create an Entity-Relationship Diagram (ERD) to plan your data model.

• Create a table that matches your ERD and load some data into the database.
• Run sample queries on the Database to handle some of the use cases.
• Launch an ElastiCache for Redis Cluster.

• Configure the Security Group on the Redis Instance. This will help us access the Instance from the Cloud9 development environment.
• Test the configuration by connecting to the Redis Instance.
• We will be using Redis Sorted Sets for fast lookups in ElastiCache.
• Load your Sample Data into multiple Sorted Sets and read the top items from the Sorted Sets.
• Create and Configure an Amazon Cognito User Pool & Client for the User Pool for User Registration, Login and Verification.

• Deploy a Lambda Function and Configure REST API with API Gateway. Invoke an Endpoint to test the application.

• Test the Application:
a. First, you start with a Registration endpoint, where a new user signs up and creates their account.
b. Second, you use a Login endpoint where a user can use a client (such as a web application or a mobile app) to authenticate and receive an ID token.
c. Third, you use a AddUserScore endpoint to record a score for a user.
d. Fourth, you use the FetchUserScores endpoint to retrieve the top scores for a particular user.
e. Finally, you use the FetchTopScores endpoint to retrieve the global top scores for the current day and month as well as the top scores of all time.