Building a Scalable Real-Time NBA Stats Pipeline With AWS: Unlocking Seamless Data Integration

NBA Statistics Pipeline🏀

🚀 Introduction

This project is an NBA Statistics Pipeline that fetches NBA team statistics from the SportsData API and stores them in AWS DynamoDB. The project also implements structured logging using AWS CloudWatch, enabling efficient monitoring and debugging.

This project was built to demonstrate my proficiency in AWS services, Python, API integrations, and Infrastructure as Code (IaC).

🛠 Tech Stack

• Python (Data processing, API requests, logging)
• AWS DynamoDB (NoSQL database for storing NBA stats)
• AWS CloudWatch (Logging & monitoring)
• Boto3 (AWS SDK for Python)
• Docker (Containerization)
• EC2 Instance (Compute environment for development)

🎯 Features

• Fetches real-time NBA statistics from the SportsData API
• Stores team stats in AWS DynamoDB
• Structured logging with AWS CloudWatch
• Error handling and logging with JSON structured logs
• Uses environment variables for sensitive credentials
• Implements batch writing for efficiency

📸 Snapshots

• API Response Sample

• DynamoDB Table Data

• CloudWatch Logs (Structured logs for monitoring)

• Terminal Output (Successful execution of the pipeline)

🏗 Project Architecture

└── nba-stats-pipeline
    ├── src
    │  ├── __init__.py
    │  ├── nba_stats.py
    │  ├── lambdafunction.py
    ├── requirements.txt      # Dependencies
    │   ├── .env              # Environment variables
    │   ├── Dockerfile        # Containerization setup (if applicable)
    ├── README.md             # Project documentation

🚀 Step-by-Step Guide to Building the NBA Stats Pipeline

4️⃣ Launch EC2 Instance and SSH Into It

ssh -i "nba-stats-pipeline.pem" [e-mail]

1️⃣ Clone the Repository

git clone https://github.com/onlyfave/nba-stats-pipeline.git
cd nba-stats-pipeline

1️⃣ Install Python3

Python3 is required to run the project.

sudo apt update
sudo apt install python3

1️⃣ Install Pip

On most systems, pip comes pre-installed with Python3. To verify, run:

pip3 --version

If you don't have pip installed, use the following command:

sudo apt install python3-pip

2️⃣ Install Dependencies

pip install -r requirements.txt

3️⃣ Set Up Environment Variables

Create a .env file with the following content:

SPORTDATA_API_KEY=your_api_key
DYNAMODB_TABLE_NAME=nba-player-stats
AWS_ACCESS_KEY_ID=your_access_key
AWS_SECRET_ACCESS_KEY=your_secret_key
AWS_REGION=us-east-1

4️⃣ CD Into the Folder Containing the Pipeline

cd src

4️⃣ Run the Pipeline

python3 nba_stats.py

📊 Sample Data Format

[
  {
    "TeamID": 1,
    "TeamName": "Los Angeles Lakers",
    "Wins": 25,
    "Losses": 15,
    "PointsPerGameFor": 112.5,
    "PointsPerGameAgainst": 108.3
  }
]

🏗 Deployment (Optional: Dockerized Version)

To run this project inside a Docker container:

docker build -t nba-stats-pipeline .
docker run --env-file .env nba-stats-pipeline

🔥 Key Takeaways

• AWS Expertise: Used DynamoDB & CloudWatch for data storage & monitoring
• DevOps Skills: Managed credentials, logging, and error handling efficiently
• Cloud-Native Thinking: Designed a cloud-based ETL pipeline

📌 Next Steps

• Implement Lambda Functions for automated execution
• Deploy using AWS ECS or Kubernetes
• Integrate with Grafana for real-time data visualization

📢 Connect With Me

🚀 LinkedIn | 🐦 Twitter/X