DEV Community: Evan

Creating a Datalake in AWS for NBA analytics!

Evan — Sun, 12 Jan 2025 04:31:39 +0000

Hey there 👋🏼 thanks for stopping by! Today, I will be teaching you how to automatically spin up an AWS S3 bucket, create a Glue database and fetch NBA data to store in the bucket. From there we will utilize AWS Athena to query the data. Stick around to learn more on AWS S3, Glue, Athena and Python!

Purpose

The purpose of this article is to walk you through how to leverage AWS tools and technologies to fetch, store and visualize data. The technologies that will be used in this article include: GitHub, AWS (S3, Athena, Glue) and Python.

Resources

Discord
YouTube Video from Alicia
My GitHub Repo

Prerequisites

VS Code (or your favorite text editor)
AWS Free Tier Account
GitHub Account
SportsData.io Account and API key
Knowledge of Git, Linux commands, APIs and AWS

Architecture

Setup

1. Clone the repo

```
git clone https://github.com/asciikeyboard/nba-datalake.git
cd nba-datalake
```

2. Log into the AWS console and launch CloudShell

3. Create the Python file

In the CLI, type nano setup_nba_data_lake.py
Copy and paste the contents from the setup_nba_data_lake.py file in the src folder in GitHub into the CloudShell terminal
Press ^X to exit, press Y to save the file, press enter to confirm the file name

4. Create the environment variables file

In the CLI, type nano .env
Paste the following info into the file and update with your API key

SPORTS_DATA_API_KEY=your_sportsdata_api_key
NBA_ENDPOINT=https://api.sportsdata.io/v3/nba/scores/json/Players

Press ^X to exit, press Y to save the file, press enter to confirm the file name

5. Install dotenv in CloudShell and run the script

Type pip install python-dotenv and hit enter
Once dotenv is installed, run the python script python3 setup_nba_data_lake.py. You should see the following output if successful:

6. Verify resources are in AWS
In IT there is a phrase that goes like this "trust but verify". That's what we are going to do now.

Search for S3 in the AWS console and check to see if your bucket is there
Next, search for Athena in the AWS console and run a query against the newly populated data:

SELECT FirstName, LastName, Position, Team
FROM nba_players
WHERE Position = 'PG';

The result should look like this:

Recap

Now, you have successfully created a Python script that automagically creates an S3 bucket and Glue database with NBA data. You then learned how to leverage Amazon Athena to visualize that data. You are well on your way to becoming a cloudy!

Creating a Subscriber Service for NBA Games using AWS technologies!

Evan — Thu, 09 Jan 2025 21:35:18 +0000

Hey there 👋🏼 thanks for stopping by! Today, I will be teaching you how to create a subscriber service in AWS to pull real-time NBA game data and send it via email or SMS. Stick around to learn more on AWS, APIs and more!

Purpose

The purpose of this article is to walk you through how to leverage AWS to create a subscriber service that utilizes an API and allows users to subscribe for NBA game information. The technologies that will be used in this article include: GitHub, AWS (Lamba, EventBridge, SNS) and Python.

Resources

Discord
YouTube Video from Ifeanyi
My GitHub Repo

Prerequisites

VS Code (or your favorite text editor)
AWS Free Tier Account
GitHub Account
Knowledge of Git, Linux commands, APIs and AWS

Architecture

Setup

1. Clone the repo

```
git clone https://github.com/asciikeyboard/gd-notifications.git
cd gd-notifications
```

2. Create AWS SNS Topic and Subscriber

Login to AWS console
Search for SNS
Select Topics in left-hand menu
- Select Create topic
- Type = Standard
- Name = gd-topic
Select Create subscription
- Protocol = email
- Endpoint = email address

It should look something like this :]

3. Create IAM Policy for SNS Topic

Open IAM in AWS console
Select Policies in left-hand menu
Select Create policy
- Service = SNS
- Change Policy editor to JSON format
- Navigate to Git repo and copy code from gd_sns_policy.json file within the policies folder
- Paste this code into the Policy editor in the AWS console
- Update the ARN on line 7 of the code with your gd-topic ARN
- Select Next
Set the Policy Name
- Policy name = gd_sns_policy
- Select Create Policy

4. Create the IAM Role SNS Topic

Navigate to IAM page
Select Roles
Select Create role
- Trusted entity type = AWS service
- Use case = Lambda
Select Next
Search for "gd_sns_policy" and select the checkbox
Search for "AWSLambdaBasicExecutionRole" and select the checkbox
Role name = gd_lamba_role
Select Create role

5. Create the Lambda function

Search for Lambda in the AWS console
Select Create function
- Select Author from scratch
  - Function name = gd_notifications
  - Runtime = Python 3.13
- Select the Change default execution role dropdown and change Execution role to Use an existing role
  - Existing role = gd_lambda_role
- Select Create function
Copy code from "gd_notifications.py" file in Git and paste into "lambda_function.py" within AWS console
Select Deploy from left-hand menu
Change from Code to Configuration menu and add the following Environment Variables:
- NBA_API_KEY = your API key
- SNS_TOPIC_ARN = your SNS topic ARN
Select Save

6. Understanding the lambda_function python script
For a detailed breakdown please reference the YouTube video from Ifeanyi

Here's an overview of what the script does: 
1. Retrieves environment variables.
2. Adjusts the current time to Central Time (UTC-6).
3. Fetches NBA game data from the external API.
4. Formats the game data.
5. Publishes the formatted data to the specified SNS topic.

7. Leverage EventBridge to run a scheduled lambda function

Search for EventBridge in the AWS console
Select Create rule
- Name = gd_rule
- Rule type = schedule
Select Continue in EventBridge scheduler
Select Recurring schedule under the Schedule Pattern option
Input Cron expression variables
- Minutes = 0
- Hours = 9-23/2,0-2/2
- Day of the month = *
- Month = *
- Day of the week = ?
- Year = *
Set Flexible time window off
Select Next
Target detail = AWS Lambda Invoke
Lambda functions = gd_notifications
Select Next
Select Next
Select Create schedule

Review

Now you have successfully created a scheduled SNS Topic to retrieve NBA game data and send that to a subscriber. Pretty neat!

Creating a Weather Dashboard using AWS S3 buckets and OpenWeather API

Evan — Mon, 06 Jan 2025 20:52:06 +0000

!!! Fair warning this is my first Dev.To post so it won't be the prettiest thing ツ

Purpose

The purpose of this post is to outline the steps I took to create a Weather Dashboard utilizing AWS, OpenWeather, GitHub, VS Code, etc.

This is a part of the #DevOpsAllStarsChallenge hosted by a few Cloud Engineers and DevOps folks that are trying to help others get involved. It's a pretty neat community and I highly recommend you take a look at the Discord if you're interested!

Resources

Discord
YouTube Video from Shae

Tools Needed

VS Code (or your favorite text editor)
AWS Free Tier Account
OpenWeather Account
GitHub Account
Knowledge of Git, Linux commands, APIs and AWS

Prerequisites

Aside from installing the necessary software and packages (such as VS Code, Python, pip, etc.), the next few things are necessary in order to properly pull from the OpenWeather API and dump that data into an AWS S3 bucket.

Create API key in OpenWeather
Create an account with OpenWeather, select "API Keys" and generate an API key. This will be used later in the .env file within the Git repo.

Create IAM user in AWS, assign group, permissions and generate Access Key
This step is imperative in order to allow the python script to function. The script goes out and creates the S3 bucket using the user credentials and the secret key. For further explanation, please reference the AWS IAM user documentation.

I have created the user "weather-data", assigned the user to a group, and given the group permissions of "AmazonS3FullAccess". This gives the user the necessary permissions to create an S3 bucket.

Spinning Up the Weather Dashboard

Clone the repo

git clone https://github.com/asciikeyboard/weather-dashboard.git
cd weather-dashboard

Create a .env file with your OpenWeather API key and AWS credentials.

touch .env

Now it is up to you to use vim, nano, or your favorite text editor to make the necessary modifications to the .env file.

Below you will see the different variables necessary. The blurred variables should never be shared as these are sensitive keys/IDs.

Run the python script

python3 src/weather_dashboard.py

The output should look similar to this:

"Trust but verify" ... so let's check AWS and boom - the data is in the newly created S3 bucket!

Now delete the S3 bucket to ensure you don't get billed!

Conclusion

This project is a nice way to get introduced to common DevOps tools such as GitHub, Python, AWS, etc. This repo allows a user to pull weather data from an API and upload it to a S3 bucket in Amazon. There are plenty of customizations that could be made. Feel free to explore!