Day 3 -> Athena empties the quiver

Michael Austin — Thu, 09 Jan 2025 18:57:51 +0000

Overview provided by: Alicia Ahl

Github Link: https://github.com/MikeAA97/DevOpsAllStarsChallenge2025/tree/main/Day3

At a high-level this day's project was based around building out an AWS Data Lake and understanding the components that make it up. I've never used AWS Glue or Athena resources before so this was a great introduction to those tools.

AWS Resources Used:

AWS Lambda: Handles the logic for querying the API, transforming the data, and pushing it to S3
AWS S3: Where both the API Data was stored, and results from Athena Queries mentioned later in the section
AWS Glue Database: A container to hold similar AWS Glue Tables / segment them for different projects or datasets.
AWS Glue Table: Defines the location of the data, as well as the schema and format, so that other tools such as Athena know how to interpret the data.
AWS Athena Workgroup: Container for Queries, as well as shared-settings. In this case I wanted to specify the output location instead of using the default.
AWS Athena Queries: A few template queries to start interacting with the dataset.

Terraform Experience

I initially struggled with managing Athena's query result output location because I was creating a new Athena database and table instead of just referencing my Glue Catalog. I realized that the database creation was aimed at manipulating the default/primary workgroup’s output location, which resulted in a dummy database. Instead of this, I opted to create a dedicated workgroup for my named_queries and specified a custom output location within the workgroup resource.

resource "aws_athena_workgroup" "output" {

    name = "output" #I'm bad with naming things

    configuration {
        result_configuration {
            output_location = "s3://${aws_s3_bucket.day3.bucket}/athena-results"
        }
    }
}



resource "aws_athena_named_query" "test-query" {

    name      = "EXAMPLE-QUERY"
    workgroup = aws_athena_workgroup.nba_analytics_output.id
    database  = aws_glue_catalog_database.day3.name

    query = "SELECT * FROM ${aws_glue_catalog_database.day3.name}.${aws_glue_catalog_table.nba_players_table.name} limit 10;"

}

This gave me better control without needing a dummy database.

Key Takeaways

Terraform versus Boto3 is going to be an interesting battle as the architecture gets more complex. I'm already seeing how I have to refactor certain api_calls to fit into a terraform-centric model.
Oddly enough, I feel like seeing the resource references within the terraform resources helps me to envision how everything relates to one another better than looking at pure Python code. It forces me to give every resource scrutiny about it's place in the workflow.

Last Thoughts

I was able to re-use a lot of the logic from Days 1 and 2 to tackle this project, which saved a ton of time. I was able to re-direct that energy to understanding what I was trying to automate.
I'll probably delay Day 4 to begin trying out integrations w/ Splunk for these previous projects, as well as work on my diagramming skills. Hopefully going forward the entire process will be more streamlined, and visually appealing.

First Post!

Michael Austin — Wed, 08 Jan 2025 05:42:23 +0000

This is my first tech-related post ever, and probably my first social-media-esque post in over 2 years!

Some thoughts about Day 2 of the 30 Day DevOps Challenge I signed up for. (I haven't fully digested Day 1 yet, but will come around to that)

Day 2 -> Finally getting emails from the NBA

Code and Architecture provided by: YouTube Video

The goal of this project was to familiarize myself with AWS tooling and working with external APIs. I utilized the SportsData.io API for NBA game information for the day. The only prerequisite for using this API is signing up for an account.

AWS Resources Used:

AWS Lambda: Handles the logic for querying the API and transforming the data into a human-readable format.
AWS SNS: Sends the data to a specified email address.
AWS EventBridge: Schedules the Lambda function invocation at set times.

Terraform Experience

To challenge myself, I decided to deploy all the AWS resources using Terraform. While I use Terraform regularly at work, I had never set it up on my personal device, so bootstrapping it was an interesting experience.

Highlights of getting Terraform set up:

Terraform Version 1.10.0: I opted for this version because it supports native state-locking in S3 buckets, which was a key feature I wanted to try out (I still use version 0.13 at work).
Installing Terraform: When I ran brew install terraform, it only installed version 1.5, which didn’t meet my needs. To resolve this, I installed tfenv, a tool that helps manage multiple Terraform versions. After installing tfenv, I ran tfenv install 1.10.0 and tfenv use 1.10.0 and was good to go.

Key Focus Areas for the Terraform Module

When converting the project from individual resources to a Terraform module, I focused on:

Clear and relevant resource naming (I’m not great at naming things)
Ensuring sensitive data was not committed to code
Making the module generic and reusable for others, with clear instructions on how to use it.

Last Thoughts

It's been great to use my personal device for a change. I've definitely taken for granted how well set-up my work environment is, so achieving some form of parity is a main focus during the beginning days of this challenge.

When time permits, I’d like to further explore the NBA data, possibly injecting it into Splunk to create visualizations.(And try not to incur any significant AWS fees in the process). It'd also be nice to try refactoring the logic into Golang to get more experience using in a different programming language in a practical-ish way.

Excited for whats in store for Day 3!

DEV Community: Michael Austin

Day 3 -> Athena empties the quiver

AWS Resources Used:

Terraform Experience

Key Takeaways

Last Thoughts

First Post!

Day 2 -> Finally getting emails from the NBA

AWS Resources Used:

Terraform Experience

Key Focus Areas for the Terraform Module

Last Thoughts