DEV Community: Matt

Static to Elastic Web App

Matt — Sat, 26 Aug 2023 15:12:39 +0000

Overview

The Web App - Single Server to Elastic Evolution is a project created by Adrian Cantrill of https://learn.cantrill.io/ which is a popular resource for people looking to earn their AWS certifications. In this project we evolve the architecture of a WordPress web app deployed on a single EC2 instance into a scalable and resilient architecture.

Stage 1 - Setup the environment and manually build WordPress

I used a CloudFormation template to set up the base 3-tier architecture which was 1 VPC with 3 AZ in us-east.

Next I manually created the EC2 to hold the WordPress instance.

Storing configuration information within the SSM Parameter store scales much better than attempting to script them in some way. In this sub-section you are going to create parameters to store the important configuration items for the platform you are building. I connected to the EC2 instance and installed WordPress.

I mapped environment variables in the EC2 to the parameters I created above. I am doing this step manually so later when I use IaC I can feel the benefit it brings.

I ran command to update OS on the instance manually and installed MariaDB, Apache web server, wget, some libraries, and a stress test utility.

I set the DB and HTTP server to run automatically when EC2 starts and set root pw to the MariaDB as the parameters we initialized earlier.

I manually downloaded and installed WordPress. Next I created the db and initialized settings manually. I went to EC2 and copied the IPv4 and logged into WordPress and made a post to this solution working.

Limitations of this implementation:

App and DB configured manually
App and DB on the same instance so they can't scale separately
Content is also stored locally in the instance so it can't scale
User connection is directly to the EC2 instance IP address for WordPress from the EC2 instance is stored statically in a DB so if I stop the EC2 instance and get assigned a new public IP I lose the original connection via the old IP

End result of architecture of this section.

Stage 2 - Automate the build using a Launch Template

I created an EC2 launch template so I don't have to manually configure the steps for setting up EC2 with WordPress and the DB in part 1.

This template can have versions which we will iterate on as time goes on
In the User Data section I can paste all the commands I used in the manual step to have it automatically initialized on startup

Same limitations as the previous stage except we automated the build of the WordPress instance using the LT.

End result of architecture of this section.

Stage 3 - Split out the DB into RDS and Update the LT

I created a subnet group so RDS can select from a range of subnets that RDS can put it's DBs inside.

3 subnets over 3 AZ

Next I created the RDS instance and migrated the data from EC2 to RDS.

To do this I connected with the session manager to the EC2 instance and exported the data from the local MariaDB

I restored the exported SQL file into my RDS instance and changed the endpoint to point to RDS.

Then I ran the SQL command to import the exported SQL file into RDS
Next I pointed the WordPress site to RDS instead of MariaDB and disable MariaDB
I went back to EC2 and open the IPv4 address and I can see the site still hosted except now it's using RDS
- To be clear though the media and WordPress post data are stored in separate DBs where the images are still in the local EC2 folder called wp-content while the post data has been migrated off EC2 to RDS
Finally I updated the launch template User Data section to remove references to MariaDB and set the new template as the default

End result of architecture of this section.

Stage 4 - Split out the WP filesystem into EFS and Update the LT

I still have the media (uploaded images) for WordPress stored locally in folder called wp-content on the EC2 instance. Next I will migrate that data to an EFS.

I created an EFS manually and added the EFS ID to the parameter store.

I went into the running EC2 instance and installed a package needed to connect to EFS.

I added a line to fstab file to mount the EFS volume to the wp-content folder each time EC2 is started.

I rebooted the EC2 instance to see if it mounts the EFS file system to it which it did successfully.

I migrated the images to the folder the EFS file system is mounted to and now EC2 is decoupled and can scale separately from the media or the posts.

Finally I updated the launch template to configure EFS in the User Data section to do the steps I did above automatically and set this version 3 of the LT as the default now.

End result of architecture of this section.

Stage 5 - Enable elasticity via a ASG & ALB and fix WordPress (hardcoded WPHOME)

Users can still connect directly to the EC2 instance and I don't have health checks or auto-healing capabilities and the IP address of the instance is hardcoded into the DB.

I created a load balancer will solve this. The LB DNS will be used in place of the EC2 IP address in order to map the DNS to each EC2 create a parameter for the LB DNS.

I created an ASG and attached it to an existing LB with a desired capacity of 1.

I enable ELB health checks and metric collection with CloudWatch

I terminated the old EC2 instance I created manually and watched the ASG automatically create an instance to match the desired amount I defined earlier.

I want to create a more dynamic scaling policy based on CPU

I created a CloudWatch alarm to go off when average CPU utilization is above 40%
I created a CloudWatch alarm to go off when average CPU utilization is below
Now I updated the ASG to have a max capacity of 3 units

I connected to the ASG EC2 and ran the stress command to test the scaling. The ASG detected high CPU so it provisioned a new EC2 instance.

I terminated this instance and watched the ASG detect this with health checks and it added another EC2 automatically to demonstrate self healing.

End result of architecture of this project.

Conclusion

Overall this was another great hands on experience. It helped to reenforce VPC and subnet concepts since I split the web app into a 3 tier architecture. It demonstrated the value having a launch template to save having to manually configuring EC2 instances. It helped me learn how to migrate data from one database to another in a different subnet and how EFS is useful for serving static files to multiple EC2 instances. Finally it demonstrated the power of ASG and ALB by making the solution elastic and enabling health checks and self healing.

CodeCommit, Build, Deploy & Pipeline using containers and ECS

Matt — Sat, 26 Aug 2023 14:49:59 +0000

Overview

The CodePipeline project is a project created by Adrian Cantrill of https://learn.cantrill.io/ which is a popular resource for people looking to earn their AWS certifications. In this project I created a CI/CD CodePipeline with CodeCommit to store static web app and Dockerfile. CodeBuild to build the Docker images and store them in ECR. ECS Cluster, TGs , and ALB configured to deploy to ECS Fargate.

Stage 1 : Configure Security & Create a CodeCommit Repo

First I connected to CodeCommit through SSH and created a repo in CodeCommit and cloned it to my local drive.

End result of architecture of this section.

Stage 2 : Configure CodeBuild to clone the repo, create a container image and store on ECR

I set up ECR to store the output of CodeBuild then I created CodeBuild project and updated role for the CodeBuild project to give it permissions to interact with ECR.

I created buildspec.yml and pushed it to the repo. The buildspec.yml file is what tells codebuild how to build your code, the steps involved, what things the build needs, and any testing and what to do with the output (artifacts).

Ran the CodeBuild project.

Now I should see a Docker image in the ECR registry I created that CodeBuild built.

I created an EC2 instance and installed Docker. Next I downloaded the Docker image to the EC2 instance from ECR that was built from the CodeBuild step.

I ran the Docker container in the EC2 instance and then checked the IPv4 of the EC2 to see if the webpage was running successfully.

End result of architecture of this section.

Stage 3 : Configure a CodePipeline with commit and build steps to automate build on commit

In this section I automated the pipeline so when code is committed to CodeCommit it is automatically built in CodeBuild and pushed to ECR. I still haven't automated deployment part though (Running the image).

I created a pipeline in CodePipeline and ran it. Success.

I updated the buildspec.yml to get the commit hash from CodeCommit and put it in a variable along with the image tag

Ran docker push with the latest tag and docker push with the image tag
Finally create an imagedefinitions.json which will be used by CodeDeploy later to deploy to ECS

I commit this new yaml file and now in the ECR I can see the latest image tagged as 'latest' along with the commit ID. If I were to push another it would remove latest from the name and just be the commit ID.

In S3 I can see the imagedefinitions.json that got created during the automatic build that I will use to automate the deploy to ECS next. It contains the name and imageUri.

End result of architecture of this section.

Stage 4 : Create an ECS Cluster, TG's , ALB and configure the code pipeline for deployment to ECS Fargate

I added deploy stage to ECS and created a ALB to attach to ECS cluster to perform scaling. Next I created an ECS cluster.

I tested the webpage is being hosted. Success. Next I added a stage to CodePipeline to deploy the image. I then modify the HTML to see if this change is propagated to the web app.

In the options when configuring select the BuildArtifact which is the S3 bucket holding the JSON file which contains the commit info and imageUri needed to deploy the built image.

The final pipeline is whenever I commit something to CodeCommit it's going to run a build and generate a docker image, store that docker image in ECR and deploy the image to the ECS service.

End result of architecture of this project.

Conclusion

Overall this was some great hands on experience to go along with the theory already I had. It helped to reenforce CI/CD concepts by breaking them down into steps. It also gave some insight into what makes ECS Fargate powerful since I didn't have to do all the EC2 management like I did initially to get the container deployed in EC2.

The Cloud Resume Challenge

Matt — Sat, 26 Aug 2023 04:24:44 +0000

Overview

The Cloud Resume Challenge by Forrest Brazeal is a project framework that guides participants as they make a resume website fully hosted in the cloud utilizing many of the different services. This isn't a step by step tutorial that shows you have to do every part but more of an assignment that tells you what to implement next but not exactly how. It requires the participants to be resourceful, make mistakes, and in the end, helps reinforce the theory they have learned in the classroom.

Part 1: Cloud Website:

In this section I set up my HTML/CSS, host it in S3, point a CDN to it and assign the CDN a custom DNS and secure it with an SSL certificate.

HTML, CSS

My HTML, CSS, and JavaScript skills are enough to be dangerous but I'm no front end dev. I found a free template and spent admittingly way too much time customizing it to the way I like before starting the main purpose of this challenge which is deploying and building on this solution in the cloud.

Set up MFA/IAM

I set up MFA using my phone on my root account and then created an IAM user to use as it's best practice to not use the root account.

I installed and set up AWS Vault on my local machine. AWS Vault is a tool to securely store and access AWS credentials in a development environment. AWS Vault stores IAM credentials in your operating system's secure keystore and then generates temporary credentials from those to expose to your shell and applications. It's designed to be complementary to the AWS CLI tools.

Static Website using S3

I wanted to set up IaC using AWS SAM now instead of the later to get practice using it to set up my services.

First I needed to install the SAM CLI.

Next I initialized a SAM template and then set the IAM policy.

Once that was created I changed to the created directory and ran sam build but It's having problems finding my python.exe

Build Failed Error: PythonPipBuilder:Validation - Binary validation failed for python, searched for python in following locations : ['C:\Users\mleve\AppData\Local\Microsoft\WindowsApps\python.EXE', 'C:\Users\mleve\AppData\Local\Microsoft\WindowsApps\python3.EXE'] which did not satisfy constraints for runtime: python3.8. Do you have python for runtime: python3.8 on your PATH?
I had to install Python 3.8.10 and checkbox the setting to Add Python 3.8 to PATH which fixed the issue

Deploy SAM with 'aws-vault exec my-user --no-session -- sam deploy --guided'

I got another error Error: Failed to create managed resources: An error occurred (InvalidClientTokenId) when calling the CreateChangeSet operation: The security token included in the request is invalid.
To resolve this I deleted my old access key and created a new one and re-initialized my-user in the aws-vault

Now if I go to CloudFormation (and switch to the correct region) I will see two new stacks created.

I want to create and S3 bucket using SAM so I added in the below command to my template.yml file in the Resources section then re-ran sam build to re-build the template then ran 'aws-vault exec my-user --no-session -- sam deploy'. No need for the --guided part of the cmd as those settings were saved in a file.

Use this in future to do both in 1 cmd: sam build && aws-vault exec my-user --no-session -- sam deploy

The next step was to add onto this config below but I kept getting ACL access errors.

Bucket cannot have ACLs setwith ObjectOwnership's BucketOwnerEnforced setting (Service: Amazon S3; Status Code: 400; Error Code: InvalidBucketAclWithObjectOwnership
- After some research it turns out "This is a legacy property, and it is not recommended for most use cases. A majority of modern use cases in Amazon S3 no longer require the use of ACLs, and we recommend that you keep ACLs disabled. For more information, see Controlling object ownership in the Amazon S3 User Guide."

Now on the properties tab I can see Static Web Hosting is enabled with this URL http://cloud-resume-evenson.s3-website.us-east-2.amazonaws.com but in the sam build I'm getting a new error "API: s3:PutBucketPolicy Access Denied".

According to this post (https://github.com/aws/aws-cdk/issues/25358) around April 2023 S3 has changed the default to ObjectOwnership: BucketOwnerEnforced, which means it is no longer possible to configure ACLs on the buckets and on objects by default.
In the amazon docs it shows what settings you need to grant public access to S3 buckets in YAML format (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket.html#aws-properties-s3-bucket--examples) which I added to my template.yml file which resolved this issue and set my S3 bucket to public by default and added my read policy correctly

CDN

Since I wanted to use IaC to build as much of this as possible instead of using the web interface I updated my template.yml file with a CloudFront distribution section and ran sam build && deploy to create the CDN. This created a CDN that I could use to access my S3 bucket through an auto generated domain name that wasn't very pretty (https://d2yhamyo6y5t0x.cloudfront.net).

Now that I can access S3 directly through the CDN and the S3 link I removed the public access I granted earlier to S3 so the only way to access the web app is through the CDN.

DNS/HTTPS

To get a nicer looking domain name I went to the Route 53 DNS and bought a domain that I can use for my website.

I also want to secure the site so I went to AWS Certificate manager to create an SSL certificate for the domain I bought.

I added a Route53Recordset to my yaml file with my updated domain name and host zone ID and mapped it back to the S3 origin name that the CDN distribution section points to.

I need to allow my custom domain access to the CloudFront distro so I need to also add a CertificateManager section with my domain name I purchased and create an SSL for to the yaml file.

Originally I created all my infra in us-east-2 and kept getting an error when trying to use SAM to attach a certificate to my CloudFront CDN. I tried creating my certificate in us-east-1 and 2, nothing seemed to work. I deleted my whole stack and restarted deploying everything in us-east-1 the next day and low and behold it works now. IaC already coming in handy.

Part 2: Serverless API

In this section the challenge is to set up the infrastructure to later add JavaScript to the front end to keep track of the number of visitors. In order to do this I set up a DynamoDB table to store the number of visits, create a POST route on my API Gateway, and write some python in a Lambda function to update the database.

Database

Next I created a DynamoDB table. Since there is no real schema I just needed to define the key schema and ID attributes.

I'll test the table once I set up the API.

API

SAM already creates an API gateway by default so I just need to define a POST method on the /visit route.

For the lambda I wrote it using Python since I'm familiar with it.

I used the get_item method to return the number of visits from the database
I then incremented the counter by 1 and used the put_item method to update the value in the database
Finally in the return section I return statusCode 200 with the visit_count in the body. I also set the CORS headers to allow all so I don't get an error since I didn't implement CORS for this project

Python

I followed an example written in Golang and translated it to Python.

I went to run a test event in lambda and got the below error. After some research it seems the error is with support for requests for urllib3 for python versions >=3.7 and <3.10. I can either specify "requests >= 2.28.2, < 2.29.0" which will use an older version of openssl in my requirement.txt or upgrade my python dependency in my yaml file to be 3.10 instead of 3.8 and on my local machine. I choose the update my requirements.txt and that fixed the issue.

"errorMessage": "Unable to import module 'app': urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'OpenSSL 1.0.2k-fips 26 Jan 2017'. See: https://github.com/urllib3/urllib3/issues/2168"

Part 3: Front End / Back End Integration

In this section I added some JavaScript to invoke the API Gateway which will trigger the Lambda function and update the count in my database. I also added some unit tests to my Python code.

JavaScript

This next section looks simple but took more time than I thought. I created an AJAX function to call my API endpoint and then update the amount in the HTML. At first I didn't realize I needed the jQuery script tag to run the AJAX code. Then I accidently made the id the same for the div and span so my "Visitors:" word was also getting removed when the AJAX was run.

Tests

I created a simple unit test to check for a 200 response code from the lambda function and then if it returned the visit count in the body as a numeric like expected.

I had to update my JavaScript to parse this object since before I was sending the raw number and now I was sending {visit_count: 60} and it was displaying as [object Object] in my html before the fix.

Part 4: Infrastructure as Code and CI/CD

In this section you were supposed to set up the IaC but I started building it from the beginning and I feel like this helped me incrementally iterate on the template as IaC is a new topic for me. I also set up a CI/CD pipeline to test, build, and deploy my infra on each commit. I did this using GitHub Actions because I was more familiar with it from previous projects and hadn't gotten a chance to use the CodePipline AWS provides.

Infrastructure as Code

I've been using AWS SAM which under the hood builds on CloudFormation since the beginning. This has come in extremely handy but also caused some headaches. There were times where I needed a fresh restart of my stack to start over at a point where I knew the code was working. In this case IaC came in very helpful as doing this all manually would have been a pain. On the other hand there were times where I didn't fully understand why I would get errors, mainly in CloudFront when trying to update my infra. This would cause me to chase down some pretty nondescriptive bugs and end up just deleting the stack and starting over adding in 1 piece at a time until I could determine what was causing the error. This was especially true when I was trying to add my lambda function using SAM. I had to narrow it down to the CodeUri path being incorrect by re-launching until I narrowed it down to that command since the error message was generic.

CI/CD

First I set up GitHub Actions by creating a .gihub/workflows/main.yml in my root directory of my project. I added an access key and secret access key from my IAM user to GitHub actions so I can reference these variables in my main.yml file.

The first test I automated was the python lambda code.

Next I added a test that depends on the first one passing that will run a sam build and sam deploy to deploy the infra.

Finally in the last block if the previous pass I automate the front end upload by having it upload the static files to the S3 bucket.

It's failing in the build and deploy infra block.

Adding --use-container to sam deploy solved this and now it works

Conclusion

As someone new to the cloud this project definitely lived up to the "Challenge" part of the name. I want to give credit to the YouTube channels "Open Up The Cloud" and "Cumulus Cycles" because without them this journey would have been much longer and more frustrating. Overall I really enjoyed this project. I feel like I grew a lot as a developer and gained confidence in my cloud skills. I plan to keep this project updated to display for years to come.