Q) What is a Modern Application?
A) • Modern applications isolate the business logic, optimizes reuse and iteration, it removes the overhead everywhere possible.
• In plain words, Modern apps are built using services that enable you to focus on writing code while automating infrastructure maintenance tasks.
• Let's look at building one such modern web application using AWS services.
Technologies used:
AWS Cloud9
Amazon Simple Storage Service (S3)
AWS Fargate
AWS CloudFormation
AWS Identity and Access Management (IAM)
Amazon Virtual Private Cloud (VPC)
Amazon Elastic Load Balancing
Amazon Elastic Container Service (ECS)
AWS Elastic Container Registry (ECR)
AWS CodeCommit
AWS CodePipeline
AWS CodeDeploy
AWS CodeBuild
Amazon DynamoDB
Amazon Cognito
Amazon API Gateway
AWS Kinesis Firehose
Amazon S3
AWS Lambda
AWS Serverless Application Model (AWS SAM)
AWS SAM Command Line Interface (SAM CLI)
• We will build a sample website called Mythical Mysfits that enables users to adopt a fantasy creature (mysfit) as pet. Here is the working sample of this website at: www.mythicalmysfits.com
• We will be walking through the steps to create an architected web application.
• We will be hosting this web application on a front-end web server and connect it to a backend database.
• We also will be setting up user authentication and will be able to collect and analyze user behavior.
• The site also provides basic functionality such as ability to “like” your favorite mysfit and reserve your chosen mysfit for adoption.
• It also allows you to gather insights about user behavior for future analysis.
The below application architecture diagram provides a structural representation of the services that make up Mythical Mysfits and how these services interact with each other.
Let's begin by hosting the static content (html, js, css, media content, etc.) of our Mythical Mysfit website on Amazon S3 (Simple Storage Service).
S3 is a highly durable, highly available, and inexpensive object storage service that can serve stored objects directly via HTTP. This makes it wonderfully useful for serving static web content directly to web browsers for sites on the Internet.
• Sign into your AWS console and select the region of your choice.
• Navigate to the Cloud9 page and create an environment with default settings.
• We will be completing all the development tasks required within our own browser using the cloud-based IDE, AWS Cloud9.
Use the below git command in the terminal to clone the necessary code to complete this tutorial:
git clone -b python https://github.com/aws-samples/aws-modern-application-workshop.git
Please, note that the above repository is a property of AWS and they own its entire copyrights.
Now, change directory to the newly cloned repository directory:
cd aws-modern-application-workshop
We are all set to create the infrastructure components needed for hosting a static website in Amazon S3 via the AWS CLI.
• Run the below command in the terminal to create the S3 bucket with the name of your choice.
aws s3 mb s3://REPLACE_ME_BUCKET_NAME
• We will be setting some configuration options that enable the bucket to be used for static website hosting.
• Run the below command in your terminal to make your S3 bucket ready for static website hosting:
aws s3 website s3://REPLACE_ME_BUCKET_NAME --index-document index.html
• By default, the public access will be blocked for the S3 bucket for security reasons.
• We need to enable the public access as this bucket is static hosting by changing the Bucket policy.
• Bucket Policy is a JSON file located at ~/environment/aws-modern-application-workshop/module-1/aws-cli/website-bucket-policy.json.
• Open the file and make the necessary changes such as entering your bucket name.
• Now, Execute the following CLI command to add a public bucket policy to your website:
aws s3api put-bucket-policy --bucket REPLACE_ME_BUCKET_NAME --policy file://~/environment/aws-modern-application-workshop/module-1/aws-cli/website-bucket-policy.json
• Now that our new website bucket is configured appropriately, let's add the first iteration of the Mythical Mysfits homepage to the bucket by running the following S3 CLI command
aws s3 cp ~/environment/aws-modern-application-workshop/module-1/web/index.html s3://REPLACE_ME_BUCKET_NAME/index.html
• Our S3 bucket is up and hosting.
• Open your favorite browser and run the below command with your inputs.
http://REPLACE_ME_BUCKET_NAME.s3-website-REPLACE_ME_YOUR_REGION.amazonaws.com
• Congratulations, you have created the basic static Mythical Mysfits Website!
• Our Mythical Mysfits website needs to integrate with an application backend. So, we will create a new microservice hosted using AWS Fargate.
• We need to create the core infrastructure environment that the service will use, including the networking infrastructure in Amazon VPC, and the AWS IAM Roles.
• To accomplish this, we will be using AWS Cloudformation.
• It is available in the location: /module-2/cfn/core.yml
• Run the below command in the Cloud9 terminal (might take upto 10 minutes for stack to be created):
aws cloudformation create-stack --stack-name MythicalMysfitsCoreStack --capabilities CAPABILITY_NAMED_IAM --template-body file://~/environment/aws-modern-application-workshop/module-2/cfn/core.yml
Navigate to the Cloudformation page in the console and wait for the Status to change to "CREATE_COMPLETE"
The output of this stack will be used a lot later. Hence, we will be saving them in a file named cloudformation-core-output.json by running the below command.
aws cloudformation describe-stacks --stack-name MythicalMysfitsCoreStack > ~/environment/cloudformation-core-output.json
Now, we will be creating a Docker container image that contains all of the code and configuration required to run the Mythical Mysfits backend as a microservice API created with Flask.
We will build the Docker container image within Cloud9 and then push it to the Amazon ECR, where it will be available to pull when we create our service using Fargate.
All of the code required to run our service backend is stored within the /module-2/app/ directory of the repository you've cloned into your Cloud9 IDE.
Docker comes already installed on the Cloud9 IDE that you've created, so in order to build the Docker image locally, all we need to do is run the following two commands in the Cloud9 terminal:
cd ~/environment/aws-modern-application-workshop/module-2/app
docker build . -t REPLACE_ME_AWS_ACCOUNT_ID.dkr.ecr.REPLACE_ME_REGION.amazonaws.com/mythicalmysfits/service:latest
You can find the account ID from the below file:
/cloudformation-core-output.json
Once docker downloads all the necessary dependancy packages, copy the image tag from the output. We will be using it in the future.
It will be in the format: 111111111111.dkr.ecr.us-east-1.amazonaws.com/mythicalmysfits/service:latest
Let's test our image locally within Cloud9 to make sure everything is operating as expected.
Run the following command to deploy the container “locally” (which is actually within your Cloud9 IDE inside AWS):
docker run -p 8080:8080 REPLACE_ME_WITH_DOCKER_IMAGE_TAG
Enter the image tag which you have saved earlier.
We will see the below output, if the container is up and running.
* Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
To test our service with a local request, we're going to open up the built-in web browser within the Cloud9 IDE that can be used to preview applications that are running on the IDE instance.
To open the preview web browser, select Preview > Preview Running Application in the Cloud9 menu bar:
Append /mysfits to the end of the URI in the address bar of the preview browser and hit ENTER:
If successful you will see a response from the service that returns the JSON document stored at /aws-modern-application-workshop/module-2/app/service/mysfits-response.json
With a successful test of our service locally, we're ready to create a container image repository in Amazon Elastic Container Registry (Amazon ECR) and push our image into it.
In order to create the registry, run the following command:
aws ecr create-repository --repository-name mythicalmysfits/service
In order to push container images into our new repository, we will need the authentication credentials for our Docker client to the repository.
Run the following command, which will return a login command to retrieve credentials for our Docker client and then automatically execute it:
$(aws ecr get-login --no-include-email)
'Login Succeeded' will be reported if the command is successful.
Using the below command for the Docker to push the image and all the images it depends on to Amazon ECR:
docker push REPLACE_ME_WITH_DOCKER_IMAGE_TAG
Now, Run the following command to see your newly pushed Docker image stored inside the ECR repository:
aws ecr describe-images --repository-name mythicalmysfits/service
Now, we have an image available in ECR that we can deploy to a service hosted on Amazon ECS using AWS Fargate. So that, the website will be publicly available behind a Network Load Balancer.
First, we will create a cluster in the Amazon ECS.
To create a new cluster in ECS, run the following command:
aws ecs create-cluster --cluster-name MythicalMysfits-Cluster
Next, we will create a new log group in AWS CloudWatch Logs for log collection and analysis.
To create the new log group in CloudWatch logs, run the following command:
aws logs create-log-group --log-group-name mythicalmysfits-logs
Since, we have a cluster created and a log group defined for where our container logs will be pushed to, we're ready to register an ECS task definition.
The input to the CLI command will be available in the below JSON File:
~/environment/aws-modern-application-workshop/module-2/aws-cli/task-definition.json
Replace the values from the image tag and from /cloudformation-core-output.json
Once you have replaced the values in task-defintion.json and saved it. Execute the following command:
aws ecs register-task-definition --cli-input-json file://~/environment/aws-modern-application-workshop/module-2/aws-cli/task-definition.json
Rather than directly exposing our service to the Internet, we will provision NLB to sit in front of our service tier.
To provision a new NLB, execute the following CLI command:
aws elbv2 create-load-balancer --name mysfits-nlb --scheme internet-facing --type network --subnets REPLACE_ME_PUBLIC_SUBNET_ONE REPLACE_ME_PUBLIC_SUBNET_TWO > ~/environment/nlb-output.json
Retrieve the subnetIds from /cloudformation-core-output.json
A new file will be created in your IDE called nlb-output.json.
We will be using the outputs in the file in later steps.
We need to create target groups for the NLB. Run the below command:
aws elbv2 create-target-group --name MythicalMysfits-TargetGroup --port 8080 --protocol TCP --target-type ip --vpc-id REPLACE_ME_VPC_ID --health-check-interval-seconds 10 --health-check-path / --health-check-protocol HTTP --healthy-threshold-count 3 --unhealthy-threshold-count 3 > ~/environment/target-group-output.json
You can find the VPC_ID value in /cloudformation-core-output.json
The output will be saved to target-group-output.json in your IDE. We will reference the output of this file in a subsequent step.
Now, we need to create a Load Balancer Listener for the NLB. Run the following command:
aws elbv2 create-listener --default-actions TargetGroupArn=REPLACE_ME_NLB_TARGET_GROUP_ARN,Type=forward --load-balancer-arn REPLACE_ME_NLB_ARN --port 80 --protocol TCP
You can find the TargetGroup and NLB ARN in the files we saved earlier while creating NLB and TargetGroup.
We need to create an IAM role that grants the ECS service itself permissions to make ECS API requests within your account.
To create the role, execute the following command in the terminal:
aws iam create-service-linked-role --aws-service-name ecs.amazonaws.com
If the above returns an error about the role existing already, you can ignore it, as it would indicate the role has automatically been created in your account in the past.
With the NLB created and configured, and the ECS service granted appropriate permissions, we're ready to create the actual ECS service.
Open ~/environment/aws-modern-application-workshop/module-2/aws-cli/service-definition.json in the IDE and replace the indicated values of REPLACE_ME.
Save it, then run the following command to create the service:
aws ecs create-service --cli-input-json file://~/environment/aws-modern-application-workshop/module-2/aws-cli/service-definition.json
Copy the DNS name you saved when creating the NLB and send a request to it using your favorite browser.
The DNS name will be in the below format:
http://mysfits-nlb-123456789-abc123456.elb.us-east-1.amazonaws.com/mysfits
You will receive a response in JSON format.
Next, we need to integrate our website with your new API backend instead of using the hard coded data that we previously uploaded to S3.
Open the below file and update the NLB URL.
/module-2/web/index.html
Run the following command to upload this file to your S3 hosted website
aws s3 cp ~/environment/aws-modern-application-workshop/module-2/web/index.html s3://INSERT-YOUR-BUCKET-NAME/index.html
Make sure, you update the Bucket name in the command before execution.
Open the website using the S3 link. Now, it is retrieving JSON data from your Flask API running within a Docker container deployed to AWS Fargate.
Now that we have a service up and running, we may think of code changes that we make to our Flask service. It would be a bottleneck for your development speed if you had to go through all of the same steps above every time you wanted to deploy a new feature to your service.
That's where we need CI/CD.
Run the following command to create another S3 bucket that will be used to store the temporary artifacts that are created in the middle of our CI/CD pipeline executions:
aws s3 mb s3://REPLACE_ME_CHOOSE_ARTIFACTS_BUCKET_NAME
Unlike our website bucket that allowed access to anyone, only our CI/CD pipeline should have access to this bucket.
Open the following JSON file and make the relevant changes according to your services from /cloudformation-core-output.json.
Once you've modified and saved this file, execute the following command:
aws s3api put-bucket-policy --bucket REPLACE_ME_ARTIFACTS_BUCKET_NAME --policy file://~/environment/aws-modern-application-workshop/module-2/aws-cli/artifacts-bucket-policy.json
We need to create a CodeCommit repository to push and store the code. Run the following command:
aws codecommit create-repository --repository-name MythicalMysfitsService-Repository
With a repository to store our code in, and an S3 bucket that will be used for our CI/CD artifacts, lets add to the CI/CD stack with a way for a service build to occur using AWS CodeBuild Project.
Replace the values in the below file and save it.
~/environment/aws-modern-application-workshop/module-2/aws-cli/code-build-project.json
Now, execute the following command to create a project:
aws codebuild create-project --cli-input-json file://~/environment/aws-modern-application-workshop/module-2/aws-cli/code-build-project.json
We will use CodePipeline to continuously integrate our CodeCommit repository with our CodeBuild project so that builds will automatically occur whenever a code change is pushed to the repository.
Open the following file, make the changes and save it.
~/environment/aws-modern-application-workshop/module-2/aws-cli/code-pipeline.json
Use the following command to create a pipeline in CodePipeline:
aws codepipeline create-pipeline --cli-input-json file://~/environment/aws-modern-application-workshop/module-2/aws-cli/code-pipeline.json
We need to give CodeBuild permission to push container images into ECR.
Locate the below file and make changes:
~/environment/aws-modern-application-workshop/module-2/aws-cli/ecr-policy.json
Save this file and then run the following command to create the policy:
aws ecr set-repository-policy --repository-name mythicalmysfits/service --policy-text file://~/environment/aws-modern-application-workshop/module-2/aws-cli/ecr-policy.json
Now, we will be moving all of the website data into DynamoDB to make the websites future more extensible and scalable.
We will be creating a DynaomDB table with a primary index and two secondary indexes
These indexes will help us to use the filter option which is used to filter the characters in the website based on their profile.
Run the below command:
aws dynamodb create-table --cli-input-json file://~/environment/aws-modern-application-workshop/module-3/aws-cli/dynamodb-table.json
A DynamoDB Table is created. We can view the details of the table from the below command:
aws dynamodb describe-table --table-name MysfitsTable
Following table will retrieve all of the items stored in the table (you'll see that this table is empty):
aws dynamodb scan --table-name MysfitsTable
We will be using the DynamoDB API BatchWriteItem command to batch insert a number of website's items into the table:
aws dynamodb batch-write-item --request-items file://~/environment/aws-modern-application-workshop/module-3/aws-cli/populate-dynamodb.json
Now, if you run the below command. You will see that there are items loaded into the table.
aws dynamodb scan --table-name MysfitsTable
Since, our data is in the table, we need to modify our application code to read from this table instead of returning the static JSON file.
To copy the new files into your CodeCommit repository directory, execute the following command in the terminal:
cp ~/environment/aws-modern-application-workshop/module-3/app/service/* ~/environment/MythicalMysfitsService-Repository/service/
Now, we need to check in these code changes to CodeCommit using the git command line client.
Run the following commands to check in the new code changes and kick of your CI/CD pipeline:
cd ~/environment/MythicalMysfitsService-Repository
git add .
git commit -m "Add new integration to DynamoDB."
git push
Now, in just 5-10 minutes you'll see your code changes make it through your full CI/CD pipeline in CodePipeline and out to your deployed Flask service to AWS Fargate on Amazon ECS.
Feel free to explore the AWS CodePipeline console to see the changes progress through your pipeline.
Finally, we need to publish a new index.html page to our S3 bucket.
You can find the file in the below location. Make the necessary changes and save the file.
~/environment/aws-modern-application-workshop/module-3/web/index.html.
Run the following command to upload the new index.html file.
aws s3 cp --recursive ~/environment/aws-modern-application-workshop/module-3/web/ s3://your_bucket_name_here/
If you revisit your website. We can find the filter option which means that the data is loading from DynamoDB.
Now, we will create a User Pool in AWS Cognito, to enable registration and authentication of website users.
Then, to make sure that only registered users are authorized to make changes in the website, we will deploy a REST API with Amazon API Gateway to sit in front of our NLB.
Execute the following CLI command to create a user pool named MysfitsUserPool
aws cognito-idp create-user-pool --pool-name MysfitsUserPool --auto-verified-attributes email
Copy the user pool unique ID from the response of the above command.
Next, in order to integrate our frontend website with Cognito, we must create a new User Pool Client for this user pool.
Run the following command (replacing the --user-pool-id value with the one you copied above):
aws cognito-idp create-user-pool-client --user-pool-id REPLACE_ME --client-name MysfitsUserPoolClient
Now, let's turn our attention to creating a new RESTful API in front of our existing Flask service.
We need to configure an API Gateway VPC Link that enables API Gateway APIs to directly integrate with backend web services that are privately hosted inside a VPC using the following CLI command:
aws apigateway create-vpc-link --name MysfitsApiVpcLink --target-arns REPLACE_ME_NLB_ARN > ~/environment/api-gateway-link-output.json
It will take about 5-10 minutes for the VPC link to be created.
A file called api-gateway-link-output.json that contains the id for the VPC Link that is being created.
You can copy the id from this file and proceed on to the next step.
The REST API is defined using Swagger. This Swagger definition of the API is located at ~/environment/aws-modern-applicaiton-workshop/module-4/aws-cli/api-swagger.json.
Open the above location and make the necessary changes for the AWS services.
Once the edits have been made, save the file and execute the following AWS CLI command:
aws apigateway import-rest-api --parameters endpointConfigurationTypes=REGIONAL --body file://~/environment/aws-modern-application-workshop/module-4/aws-cli/api-swagger.json --fail-on-warnings
Copy the response this command returns and save the id value for the next step:
Now, our API has been created, but it's yet to be deployed anywhere.
We will call our stage prod. To create a deployment for the prod stage, execute the following CLI command:
aws apigateway create-deployment --rest-api-id REPLACE_ME_WITH_API_ID --stage-name prod
Now, the API is available in the internet. Use the following link:
https://REPLACE_ME_WITH_API_ID.execute-api.REPLACE_ME_WITH_REGION.amazonaws.com/prod/mysfits
Now, we need to include updated Python code for your backend Flask web service, to accommodates the new functionality of the website.
Let's overwrite your existing codebase with these files and push them into the repository:
cd ~/environment/MythicalMysfitsService-Repository/
cp -r ~/environment/aws-modern-application-workshop/module-4/app/* .
git add .
git commit -m "Update service code backend to enable additional website features."
git push
The service updates are being automatically pushed through your CI/CD pipeline.
Open the location /environment/aws-modern-application-workshop/module-4/app/web/index.html which contains the new version of the website's index.html file.
Make the relevant changes in the additional two html files along with index.html and save the files.
Run the below command:
aws s3 cp --recursive ~/environment/aws-modern-application-workshop/module-4/web/ s3://YOUR-S3-BUCKET/
Refresh the Mythical Mysfits website in your browser to see the new functionality in action
We will be creating a way to better understand how users are interacting with the website.
We will create a new and decoupled service for the purpose of receiving user click events from the website. This full stack has been represented using a provided CloudFormation template.
Let's create a new CodeCommit repository where the streaming service code will live:
aws codecommit create-repository --repository-name MythicalMysfitsStreamingService-Repository
Copy the value of 'cloneUrlHttp' from the above output.
Next, let's clone that new and empty repository into our IDE:
cd ~/environment/
git clone {insert the copied cloneValueUrl from above}
We will be moving our working directory into this new repository:
cd ~/environment/MythicalMysfitsStreamingService-Repository/
Let's copy the new application components into this new repository directory:
cp -r ~/environment/aws-modern-application-workshop/module-5/app/streaming/* .
And finally let's copy the CloudFormation template for this module as well.
cp ~/environment/aws-modern-application-workshop/module-5/cfn/* .
Run the below command to install the requests package and its dependencies locally alongside your function code:
pip install requests -t .
Open the streamProcessor.py file and replace the values of the ApiEndpoint with the actual value.
Now, Let's commit our code changes to the new repository so that they're saved in CodeCommit:
git add .
git commit -m "New stream processing service."
git push
Let's create an S3 bucket where AWS SAM CLI will package all of our function code, upload it to S3, and create the deployable CloudFormation template.
aws s3 mb s3://REPLACE_ME_BUCKET_NAME
Now that our bucket is created, we are ready to use the SAM CLI to package and upload our code and transform the CloudFormation template.
sam package --template-file ./real-time-streaming.yml --output-template-file ./transformed-streaming.yml --s3-bucket replace-with-your-bucket-name
Dont forget to replace the above value with the bucket name.
Let's deploy the Stack Using AWS CloudFormation. Run the below command:
aws cloudformation deploy --template-file /home/ec2-user/environment/MythicalMysfitsStreamingService-Repository/cfn/transformed-streaming.yml --stack-name MythicalMysfitsStreamingStack --capabilities CAPABILITY_IAM
Once this stack creation is complete, the full real-time processing microservice will be created.
With the streaming stack up and running, we now need to publish a new version of our website frontend that includes the JavaScript that sends events to our service whenever a profile is clicked by a user.
Below, is the location of the new index.html file:
~/environment/aws-modern-application-workshop/module-5/web/index.html
Make the relevant changes in the above file and perform the following command for the new streaming stack to retrieve the new API Gateway endpoint for your stream processing service:
aws cloudformation describe-stacks --stack-name MythicalMysfitsStreamingStack
Finally, replace the final value within index.html for the streamingApiEndpoint and run the following command.
Refresh your Mythical Mysfits website in the browser and you have a site that records and publishes each time a user clicks on a profile.
To view the records that have been processed, check the destination S3 bucket created as part of your MythicalMysfitsStreamingStack.
Be sure to delete all the resources created in order to ensure that billing for the resources does not continue for longer than you intend.
Top comments (0)