#CloudGuruChallenge – Covid-19 Event-Driven Python on AWS - Completed!

tbhagat — Fri, 02 Oct 2020 18:15:05 +0000

I completed the Cloud Resume Challenge posted by Forrest a couple months ago and when I saw this posted I knew I had to give it a try. On paper this challenge seemed a bit easier than the last but I ended writing more code and learning even more throughout it. Below I'll explain how I worked through each step of the challenge and my thought process through it, so without further ado let's get into it.

The main challenge page can be found here

The finished product has be seen here

Repo for this project GitHub

My LinkedIn Tejas Bhagat

IaC, Source Control, CI/CD

I found it easier in the last challenge to make improvements and changes after the above was setup so I went ahead and started with this. My IaC template was basic at first with just my lambda and DynamoDB but as I worked through each step it was easier to write a couple lines in my yaml template to add the other services as needed.
I wanted to make the dashboard public so when I got to creating the dashboard I ended up including another lambda function and API gateway as well.

Extraction

To extract the data I used Pandas to read the data from the New York Times and John Hopkins Github sources and pass the data in to my Transformation function.

Transformation

Here I again used Pandas, this time to convert the data to data frames and then formatting, filtering and joining for my needs. The John Hopkins data was filter to only show U.S data and then I merged the Recovered data column to the cleaned New York Times data. At one point I had the dates converted to Date objects but I found that I could not load Date Objects into DynamoDB and loaded them as strings instead.
I also Included data validation in this phase to make sure the Dates were in correct format and Deaths, Cases, Recovered were all integers and greater than 0.

Load

Once the data was cleaned I was able to load it all into my using DynamoDB by using boto3. Here I have the new data date field check against what is already in the database and if it already exists it does not load it. If the data does not exist in the database it would be loaded and a SNS notification is sent to subscribers of how many dates were added as well as the cases/deaths/recovered for each date.
Throughout the extraction, transformation and load process I have try: and except: code blocks which would exit the lambda and send subscribers a notification for errors which occur in the process. One of the most difficult steps of the challenge was writing tests for me, I was able to write a couple tests to check for malformed data and it does indeed throw an error and send the SNS message.

Dashboard

Once I had all the data in my database I was thinking of ways to make it public and add it to my site created in the last challenge. I remembered I did something similar with the visitor counter during the Resume challenge using API gateway, Lambda and DynamoDb so I decided to try that here as well.
I added another lambda which pulls the current data from the database and hooked it in with API gateway so I can use it on my site. Now that I was able to reach the data through my API I had to figure out how to use, I did some research and ended up going with Google Charts. Here I wrote some JavaScript and fiddled with HTML until the product was what I imagined.

Final architecture for this project.

In the future I’d like to try to reduce the load times of the charts and make it more interactive for the end user so they are able to see exactly what they need.

Overall this was a very fun project where I was again able to learn a lot, what I learned here will let me think bigger and better for my future personal projects and future challenges. Huge thank you to Forrest and A Cloud Guru for these challenges!

Road to the Cloud Resume Challenge.

tbhagat — Mon, 22 Jun 2020 19:29:16 +0000

I found this challenge on r/AWSCertifications after passing my CSAA test and thought it would be a great way to get more hands on exposure with AWS and to buff up my resume a bit. Though there were a lot of steps I didn't completely understand at first, I had a ton of fun googling through any roadblocks I faced and reading through pages (and pages) of documentation on various services to complete the challenge.

I ended up splitting the challenge into 3 parts, the front end (Steps 2-6), CI/CD pipeline and then back end (steps 7 -12). I started with the CI/CD early and then tacked the test of the challenge as I thought this would be the easiest way to make changes throughout the entire challenge, this held true for the front end but not as great for the back. Soo that's enough rambling, I'll get into how I tackled steps.

Steps 2-6..

I began with creating a public bucket in S3, enabling Static Website hosting in the properties and creating the public bucket policy where everyone can GET objects from the bucket but cannot make any other changes to it. I uploaded a basic index.html doc with some test html and that gets us live on the internet (although from a very ugly endpoint)

Since I already had a domain name with Route53 I created the cloudfront distribution next. Within cloudfront I set the origin to my s3 bucket, received an SSL certificate for my domain from AWS Certificates Manager and also enabled HTTP to HTTPS redirect. Going back into Route53 I created Alias records so my Domain name points to the cloudfront distribution (which also has an ugly endpoint). After completing these steps, browsing to tejasbhagat.com allowed me to see my test html. I kept this test html there until I could set up the CI/CD and update the site straight from my text editor (atom) with a couple clicks.

Steps 11-15..

Setting Up CI/CD (front end)

I used Github Actions to set up the pipeline for the front and back end. I was able to find the actions I needed on the github market place and edited them/used them as a template for my own needs. The action itself spins up a docker container and runs a series of aws cli commands which uploads files to the s3 bucket and invalidates the cloudfront cache when a push is made to the master branch. With this I was able to build out the HTML, CSS and JS for the site, view it locally and push it to prod with just a couple clicks.

Setting Up CI/CD (back end)

For the back end I set up the CI/CD pipeline first and used it to build out the API Gateway, Lambda, and DynamoDB. This was a big learning experience because I ran into countless errors during the sam build and sam deploy steps. Ultimately it came down to jumping into the rabbit hole of errors and researching/fixing (often). As of typing this out I am still working on the Python unit testing portion and integrating it with the build.

Steps 7-10..

This set of steps was also a huge learning experience. Now that the infrastructure was setup from the CI/CD I had to make changes to the initial lambda function I used so that it would properly make changes to the dynamoDB and then return the information through API gateway. I used python and boto3 to create a lambda function which increments the dynamoDB by 1 and then serves the new count every time a request is made to the API gateway endpoint. I then wrote a couple lines of javascript on my front end to pull and serves this info on my website.

Overall, I thought this challenge was a very fun experience and a great way to gain exposure to a handful of AWS services. I highly recommend it if you want to get hands on and learn more about AWS. And of course huge thank you to Forrest Brazeal for putting this challenge together!

See the final product here
https://tejasbhagat.com/