Hey there 👋🏼 thanks for stopping by! Today, I will be teaching you how to automatically spin up an AWS S3 bucket, create a Glue database and fetch NBA data to store in the bucket. From there we will utilize AWS Athena to query the data. Stick around to learn more on AWS S3, Glue, Athena and Python!
Purpose
The purpose of this article is to walk you through how to leverage AWS tools and technologies to fetch, store and visualize data. The technologies that will be used in this article include: GitHub, AWS (S3, Athena, Glue) and Python.
Resources
Discord
YouTube Video from Alicia
My GitHub Repo
Prerequisites
VS Code (or your favorite text editor)
AWS Free Tier Account
GitHub Account
SportsData.io Account and API key
Knowledge of Git, Linux commands, APIs and AWS
Architecture
Setup
1. Clone the repo
```
git clone https://github.com/asciikeyboard/nba-datalake.git
cd nba-datalake
```
2. Log into the AWS console and launch CloudShell
3. Create the Python file
- In the CLI, type
nano setup_nba_data_lake.py - Copy and paste the contents from the setup_nba_data_lake.py file in the src folder in GitHub into the CloudShell terminal
- Press ^X to exit, press Y to save the file, press enter to confirm the file name
4. Create the environment variables file
- In the CLI, type
nano .env - Paste the following info into the file and update with your API key
SPORTS_DATA_API_KEY=your_sportsdata_api_key
NBA_ENDPOINT=https://api.sportsdata.io/v3/nba/scores/json/Players
- Press ^X to exit, press Y to save the file, press enter to confirm the file name
5. Install dotenv in CloudShell and run the script
- Type
pip install python-dotenvand hit enter - Once dotenv is installed, run the python script
python3 setup_nba_data_lake.py. You should see the following output if successful:
6. Verify resources are in AWS
In IT there is a phrase that goes like this "trust but verify". That's what we are going to do now.
- Search for S3 in the AWS console and check to see if your bucket is there
- Next, search for Athena in the AWS console and run a query against the newly populated data:
SELECT FirstName, LastName, Position, Team
FROM nba_players
WHERE Position = 'PG';
The result should look like this:
Recap
Now, you have successfully created a Python script that automagically creates an S3 bucket and Glue database with NBA data. You then learned how to leverage Amazon Athena to visualize that data. You are well on your way to becoming a cloudy!
Top comments (0)