DEV Community: Jay Viloria

Event-Driven Python ETL: ACloudGuru September 2020 Challenge

Jay Viloria — Thu, 01 Oct 2020 20:32:37 +0000

I just completed the ACloudGuru September 2020 challenge but in reality I feel like I have only reached one ending of a Choose-Your-Own-Adventure game.

Basically, the challenge is to create an automated ETL process (ran once daily) that takes two COVID-19 data sources, merge and clean them, apply some transformations and save the result to a database of our choosing and send notifications about the results of the process. A dashboard was then required that used the post-ETL data as a source. Here is the official challenge announcement.

Although there were many individual steps listed in the instructions, my approach can be broken into the following parts which I'll expand on in later sections:

Analysis: Time spent exploring data sources, considering different solution routes
CI/CD
ETL Construction: Developing the Extract/Transform/Load logic
DB Choice: Choosing the right DB and how I went with a csv file and S3 as my "database"
Notifications: SNS Notification set-up...and then tear-down...and then re-set-up...
ETL/Unit/Smoke Testing: Manually testing process, writing unit tests and trying to do a proper smoke test
Data Consumption/Dashboard

Analysis

I spent a fair bit of time before doing any "work" just thinking about how I wanted to approach the challenge. I read the instructions a few times and took a look at the data. I'll list some of my initial thoughts/observations and later relate them to future choices,

After looking over the QuickSight and Tableau Public sites, I made the decision to do the frontend/dashboard myself. I wasn't sure at that time if I was going to use a framework or simply vanilla JS. I was leaning towards React though.
As far as CI/CD of resources, I have been doing a lot with GitHub Actions so I decided to continue using that, instead of the Code* line of products that AWS offers.
After glancing at the initial data, I knew I wanted to add fields, specifically, the incremental data. The existing fields were all cumulative cases/deaths/recoveries. I also noticed that at times, older data had their records updated (adjustments to cases/deaths/recoveries)
My initial impressions were that I wanted a relational DB, although cost/complexity was a big concern. This was one of the considerations that made the analysis step take longer than expected.
Forrest listed some extra challenging steps, one of them being a smoke test that ran the process with dummy data and verified the notification message. I knew I wanted to include this in my solution.

CI/CD

After deciding on using GitHub Actions for my workflow, I split my project into two repositories: back-end/ETL/Data API and front-end/dashboard.

ETL/API

I used SAM to handle the back-end. Because I decided to do the front-end myself, I didn't have native integrations into a dashboarding software/service so I knew I needed an API for the data to be consumed. I considered splitting the API into a separate SAM project but decided against it. My rationale being that for a hypothetical ETL change, for example, adding a data source, the API would probably be changed as well to make the new data available to the user.

Dashboard

By this time, I decided to go with React for the dashboard so I chose CDK to structure my project/provision my resources. This was a choice of convenience because I had just completed a template for a React project that is deployed using CDK. You can find the template repo here.

ETL Construction

This was a fairly straight-forward portion of the project. I split the Extract/Transform/Load steps into separate .py files and had one lambda handler control everything. I used one lambda for the core ETL process, but I considered an alternative approach using multiple de-coupled parts for the Extract/Transform/Load steps. In the end, I stuck with the one lambda approach mostly out of simplicity but I'll briefly describe an alternative.

Alternative Approach

I considered having an Extract lambda save the downloaded data to a "raw data" S3 Bucket, that triggered the Transform/Load lambdas. The Transform lambda would have needed logic to ensure that all data sources were extracted before doing its thing. I considered this approach when I thought about scalability of one ETL Lambda. I thought about the case where the data sets were larger or data sources more numerous. In that case, the Extract step might be much longer and the penalty for a failure in the Transform/Load steps much more severe (restarting downloads).

By that same logic, I considered a separate Transform lambda/process. This is because as the data gets bigger, the Transform process could be beyond the Lambda capabilities (15 minute run time and memory constraints) and would possibly need a Fargate/EC2 process to complete.

AWS Glue

I didn't have experience with AWS Glue but it seemed like a viable option, especially with my "database" choice described later. Part of me wants to "choose another path" and re-do an alternate version of this project using Glue instead.

DB Choice

I spent a lot of time going over this choice. My knee-jerk reaction when first looking at the data was a relational DB. This feeling was strengthened by my desire to add fields as mentioned in the Analysis section.

Unfortunately, I looked into RDS and it was expensive as well as being more complex to integrate with AWS Lambda. The main blocker was the cost though, to incur the costs associated with standing up a RDS instance for 200-300 records seemed inappropriate. I briefly looked into Aurora Serverless but I wasn't sure how scaling down to/scaling up from 0 ACU worked. This was a concern because my data was to be exposed as an API Gateway/Lambda for consumption so I wasn't sure how it would affect latency.

My other choices were DynamoDB and the alternative that I ended up choosing, a csv file stored in an S3 Bucket. I'll describe what went into my choice below,

Daily Data Diff

I noticed during my initial analysis that when the data was updated, at times, older records were updated/adjusted. The DataFrame object has nice methods tools to find the global differences between two objects. pandas allows for a csv file to be converted to a DataFrame as one operation. With DynamoDB, the comparisons have to be done at the record/item level or I suppose the result of a scan could be converted to a DataFrame. Either way, it seemed as if more operations were necessary.

Data Consumption Flexibility

My initial leaning towards a relational DB was because of the ability to query the data flexibly. In terms of scalability, I considered the case where this project was extended to include data at the state or county level or many more fields were added. In that case, when consuming the data, I believe it should be filtered on the server(less) side first. Also flexible queries in DynamoDB require setting up Secondary Indexes so I didn't like it as a choice.

I felt that a csv file stored in S3, combined with a Lambda using the S3 Select functionality gave me the best options:

In the case of new fields, there is a clear way to add new query flexibility in terms of query string parameters in the API endpoint.
If the amount of data scaled up allowing for different segmentations (country, state/province, county) and new consumers of this data, the data consumed by existing users would be relatively predictable because with S3 Select, you pay for the data after filtering takes place.
If the amount of data scaled up enough to facilitate a migration to an RDS instance, the S3 Select logic should be mostly re-usable.

Notifications

The DB choice was the most difficult choice I made in this project, but the choice on how to implement ETL notifications was by far the most frustrating choice.

At first it seemed so simple, Lambda has a Destinations feature that allows automatic integration with a few different AWS services, one of them being SNS, for asynchronous Lambda invocations. I thought it was perfect, I'll just hook that up and I get automatic success/failure publishing to my subscribers (my email address). Then I saw the notification...

I looked into the Destinations documentation and the above was what I was stuck with (Note, some content was blocked out from the screenshot above). I understand that Destinations are probably meant more for machine triggers of services so sending over a JSON-structured object makes more sense. For a human-readable notification though, it was...less than ideal. For a while, I thought, well technically, it meets the requirements of the challenge, but it looks terrible and it makes the smoke test much more difficult... But it seemed so easy to just leave that Destinations configuration in my SAM template, oh well, time to move on.

I broke up the ETL Lambda-SNS connection and instead configured the Destination of my ETL Process to be another Lambda function responsible for publishing to SNS. This accomplished two main things,

Better Structured/Formatted Notifications

Filtered Notifications

I was able to use the SNS message attributes to filter messages to subscribers. I used this to create a subscriber that only received messages during my smoke test run using an environment parameter to denote testing vs production runs.

I used an SQS Queue as the target of my testing notifications because it allowed for programmatic verification of received messages and was a direct source of the message. What I mean by that is I considered using another lambda for the test messages and writing the message content to S3/DynamoDB to verify the SNS notifications were functioning. In that sense though, I am also testing the Lambda -> {S3, DynamoDB} connection which I felt muddied the waters of the test, so to speak. This is one area where I would like to learn what is standard practice because my solution felt very hacky.

ETL/Unit/Smoke Testing

By this time I had three Lambda functions:

Core ETL Function
SNS Publishing Function
API Gateway Proxy Function

Most of my unit tests were focused on the ETL function, I tested the Extract/Transform/Load modules separately and tried to cover different data problems that might come up.

One unfortunate consequence of using S3/S3 Select as my database/data consumption mechanism is that moto, the standard mocking library for AWS services, does not yet have support for it so I couldn't include a proper unit test for my API Gateway Proxy function.

Smoke Test

Based on all the machinery I set up around the smoke test, e.g. SNS Message Attributes, SQS Queue, SNS Lambda Publisher. The actual smoke test script was very straight forward. It invokes the ETL Lambda with a testing environment value. It then polls an SQS Queue to determine if the correct SNS message was sent, then cleans up any test ETL files created in the "database". In this case, I used S3 prefixes to separate production vs testing data.

Data Consumption/Dashboard

For this part of the project, I put together a quick React app and used the Recharts library to create a graph out of the data returned by the API call I set up above.

I included a few controls to toggle how the data is aggregated or whether or not the user is viewing Cases/Deaths/Recoveries. Although initially I wanted to add filters for some of the fields I added during the Transform process, I left them as possible future enhancements, translation, I got a little tired and wanted to wrap up this project. A screenshot of the dashboard is provided

Real-Time Dashboard Updates

I didn't implement this in my project but it was listed as an extra challenging step in the directions so I thought I would describe how I might approach that.

Since both data sources were in public repos, I think I would create a function that hits the GitHub API endpoint for both sources with some level of frequency and does a diff comparison of those specific files. If there are changes to those files, trigger the ETL event.

This wouldn't be technically be real-time but with a frequent enough period of the GitHub API endpoint, it could be near it. It would also avoid downloading the file(s) unless necessary. Although I'm not sure if there are quota restrictions on the GitHub API.

Conclusion

I very much enjoyed this challenge. I still feel like there were a bunch of different ways of accomplishing certain steps and I am very tempted to do multiple iterations of everything.

Feel free to check out my dashboard or the two repos that make up the project: front-end and back-end

The overall back-end architecture is provided below,

Overall, if this is what we can expect from the ACloudGuru Monthly Challenges, I am very much looking forward to them.

P.S. I apologize for the length of this post, I just sat down and it all poured out.

Deploy React Site Using GitHub Actions & AWS CDK

Jay Viloria — Tue, 15 Sep 2020 19:19:37 +0000

My Workflow

This is a React template that uses a GitHub Actions Workflow to deploy static files to AWS in multiple development environments with a custom domain and HTTPS. The GH Actions Workflow leverages AWS CDK to create resources/provision services necessary for deployment.

The workflow assumes a Release Flow-like branching strategy that allows for the following development workflow:

automated testing for "feature/*" branch pushes
automated testing/dev deployments on pushes and pull requests into master
automated testing/stg deployments/artifact storage upon tag pushes
manual workflow for prod deployments where tagged version is specified and deployed from artifact storage

Submission Category:

DIY Deployments

Yaml File or Link to Code

jviloria96744 / react-aws-cdk-template

React JS template using AWS Cloud Development Kit (CDK) to deploy static site to AWS with a custom domain name.

React/AWS CDK Template

This project is a template used to deploy a static React site to an Amazon S3 Bucket.

The GitHub Actions Workflow is responsible for the CI/CD portion of this project and leverages the AWS Cloud Development Kit to create the S3 Bucket, CloudFront Distribution, SSL Certificate and associated permissions/access policies and will deploy to multiple sub-domains given a custom domain that the user owns.

Get Started

Clone/Fork Project

Within your own GitHub account, create a repo using a cloned local repository or by forking this repository. Clone the repository using the following command,

git clone https://github.com/jviloria96744/react-aws-cdk-template.git

GitHub Secrets/AWS Prerequisites

The GitHub Actions Workflow requires four secrets that are all associated with prerequisite activities that need to be completed in AWS (AWS Account Set Up Tutorial):

AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY
AWS_DOMAIN_NAME
AWS_REGION

The Access Key ID and Secret Access Key are the credentials of an AWS 'User' that…

View on GitHub

Cloud Resume Challenge: My Experience & My New Addiction to Green Check Marks

Jay Viloria — Thu, 23 Jul 2020 04:20:43 +0000

It was the latter half of May that my journey down the AWS rabbit hole began. I had just left my job as a Econ/Data Analyst turned Software Engineer doing web development and wanted to commit myself to:

Learning desirable skills for my market
Getting to some side projects that had been sitting collecting dust
Getting hungry again in terms of wanting to do development and build things

AWS seemed to fit nicely for covering all three points. Over the next 6 weeks, I focused on two parallel efforts: learning AWS by way of certifications (Cloud Practitioner and Solutions Architect-Associate) and making ground on my side projects using any AWS services I could take advantage of. Over that time, while trying to look for resources to learn, I came across the Cloud Resume Challenge a few times but didn't think to attempt the challenge. Once I attained the SA-Associate certification, I decided to revisit the Cloud Resume Challenge for the following reasons:

I thought it was a good time to put together a portfolio and get back out on the market
Because of the parallel nature of my approach, some of the hands-on AWS experience I had gained definitely didn't reflect best practices. I liked that the Cloud Resume Challenge required use of SAM which I had been wanting an excuse to learn.

The Challenge is conceptually straight-forward; create a portfolio/resume website, hosting it through AWS with a back-end that keeps track of visitors to the page, all using proper Source Control/CICD Practices. I'll describe my experience roughly in the order in which I did things.

Front End Resume Website

Part of my reason for taking on the challenge was to put together a portfolio so I actually spent a good amount of time, relatively speaking, on this portion (although looking at the finished product, it may not seem like it, as a designer I am definitely not). What I most liked about this portion was the bare-bones nature of the website. I had web development experience but had definitely gotten used to relying on frameworks such as Material UI or Bootstrap. It was nice doing everything with only HTML and my own CSS.

Hosting/DNS/HTTPS

As part of one of my side projects, I had gone through the AWS tutorials on S3 Static Site Hosting behind a CloudFront Distribution and Route 53 Domain Name so this part was relatively quick and straightforward. Most of it was just avoiding the stumbling blocks I ran into while doing this with my side projects, such as making sure the certificate is based in us-east-1 and getting all my ducks in a row with regards to redirecting non-root to root domains through CloudFront distributions and making sure everything was secure.

Back-End Visitor Count

Most of my side projects/AWS dabbling was done using serverless pieces like Lambda and DynamoDB. To be honest, this was more due to cost than any other architectural considerations. Fortunately though, that experience helped make the Python/JavaScript work needed to get the back-end working not anything I hadn't seen before.

Infrastructure as Code & CI/CD

This was the area where I felt I learned the most and the area that really hit that third bullet point in my initial list of goals for wanting to learn AWS. In my side projects, I admit (embarassingly) that there was definitely a lot of Lambda Function code editing going on in the console. In my mind I knew it wasn't best practices but I almost treated it like a rite of passage in a sense; some type of hazing process I needed to go through to properly see the benefits of CloudFormation and SAM. As soon as I got my first Stack built based on my commit through my GitHub Actions Workflow (which took much less effort than I was expecting), I thought to myself, "Why wasn't I doing everything this way sooner?" I am now very excited to dive deep into CloudFormation/SAM and look into CDK which from what I have read, I think I could be very on-board with. I can't wait to see more green check marks and circles.

Final Thoughts

This project was a nice challenge that forced me to update my resume/portfolio and gave me a proper push into the world of Infrastructure as Code that I don't think I'll be leaving any time soon. A huge thank you to Forrest Brazeal for putting together this challenge.