DEV Community: AdesegunA

NCAA Game Highlights

AdesegunA — Mon, 03 Feb 2025 14:48:42 +0000

Highlight Processors

This project utilizes RapidAPI to fetch NCAA game highlights within a Docker container and leverages AWS MediaConvert to process and convert the media files.

File Overview

The config.py script carries out the following functions:

It imports essential environment variables and maps them to Python variables, assigning default values when necessary.
This method ensures flexible configuration management, allowing different settings for various environments (e.g., development, staging, production) without altering the source code.

The fetch.py script performs the following actions:

Establishes the date and league that will be used to find highlights. We are using NCAA in this example because it's included in the free version.
This will fetch the highlights from the API and store them in an S3 bucket as a JSON file (basketball_highlight.json)

The process_one_video.py performs the following actions:

Connects to the S3 bucket and retrieves the JSON file.
Extracts the first video URL from within the JSON file.
Downloads the video fiel from the internet into the memory using the requests library.
Saves the video as a new file in the S3 bucket under a different folder (videos/)
Logs the status of each step

The mediaconvert_process.py performs the following actions:

Creates and submits a MediaConvert job
Uses MediaConvert to process a video file - configures the video codec, resolution and bitrate. Also configured the audio settings
Stores the processed video back into an S3 bucket

The run_all.py performs the following actions:
Runs the scripts in a chronological order and provides buffer time for the tasks to be created.

The .env file stores all over the environment variables, these are variables that we don't want to hardcode into our script.

The Dockerfile performs the following actions:
Provides the step by step approach to build the image.

The Terraform Scripts:
These scripts are used to created resources in AWS in a scalable and repeatable way. All of the resources we work with like S3, creating IAM user roles, elastic registry service and elastic container services is built here.

Prerequisites

Before running the scripts, ensure you have the following:

1. Create Rapidapi Account

Rapidapi.com account, will be needed to access highlight images and videos.

For this example we will be using NCAA (USA College Basketball) highlights since it's included for free in the basic plan.

Sports Highlights API is the endpoint we will be using

2. Verify prerequites are installed in your Local Environment

Ensure Docker should is pre-installed in most regions. you can check by typing docker --version
Ensure AWS CloudShell has AWS CLI pre-installed.you can check by typing aws --version
Ensure Python3 is pre-installed also python3 --version

3. Retrieve AWS Account ID

Copy your AWS Account ID Once logged in to the AWS Management Console Click on your account name in the top right corner You will see your account ID Copy and save this somewhere safe because you will need to update codes in the labs later

4. Retrieve Access Keys and Secret Access Keys

You can check to see if you have an access key in the IAM dashboard
Under Users, click on a user and then "Security Credentials"
Scroll down until you see the Access Key section
You will not be able to retrieve your secret access key so if you don't have that somewhere, you need to create an access key.

Technical Diagram

Project Structure

src/
├── Dockerfile
├── config.py
├── fetch.py
├── mediaconvert_process.py
├── process_one_video.py
├── requirements.txt
├── run_all.py
├── .env
├── .gitignore
└── terraform/
    ├── main.tf
    ├── variables.tf
    ├── secrets.tf
    ├── iam.tf
    ├── ecr.tf
    ├── ecs.tf
    ├── s3.tf
    ├── container_definitions.tpl
    └── outputs.tf

INSTRUCTIONS

Step 1: Clone The Repo

git clone https://github.com/adesegunasunmo/gaming-highlight-processors.git
cd src

Step 2: Add API Key to AWS Secrets Manager

aws secretsmanager create-secret 
    --name my-api-key 
    --description "API key for accessing the Sport Highlights API" 
    --secret-string '{"api_key":"YOUR_ACTUAL_API_KEY"}' 
    --region us-east-1

Step 3: Create an IAM role or user

In the search bar type "IAM"

Click Roles -> Create Role

For the Use Case type "S3" and click next

Under Add Permission search for AmazonS3FullAccess, MediaConvertFullAccess and AmazonEC2ContainerRegistryFullAccess and click next

Under Role Details, type "HighlightProcessorRole" as the name

Select Create Role

Find the role in the list and click on it
Under Trust relationships
Edit the trust policy to this:
Edit the Trust Policy and replace it with this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "ec2.amazonaws.com",
          "ecs-tasks.amazonaws.com",
          "mediaconvert.amazonaws.com"
        ],
        "AWS": "arn:aws:iam::<"your-account-id">:user/<"your-iam-user">"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Step 4: Update .env file

RapidAPI_KEY: Ensure that you have successfully created the account and select "Subscribe To Test" in the top left of the Sports Highlights API
AWS_ACCESS_KEY_ID=your_aws_access_key_id_here
AWS_SECRET_ACCESS_KEY=your_aws_secret_access_key_here
S3_BUCKET_NAME=your_S3_bucket_name_here
MEDIACONVERT_ENDPOINT=https://your_mediaconvert_endpoint_here.amazonaws.com

aws mediaconvert describe-endpoints

MEDIACONVERT_ROLE_ARN=arn:aws:iam::your_account_id:role/HighlightProcessorRole

Step 5: Secure .env file

chmod 600 .env

Step 6: Locally Buikd & Run The Docker Container

Run:

docker build -t highlight-processor .

Run the Docker Container Locally:

docker run --env-file .env highlight-processor

This will run fetch.py, process_one_video.py and mediaconvert_process.py and the following files should be saved in your S3 bucket:

Optional - Confirm there is a video uploaded to s3:///videos/first_video.mp4

Optional - Confirm there is a video uploaded to s3:///processed_videos/

What We Learned

Working with Docker and AWS Services
Identity Access Management (IAM) and least privilege
How to enhance media quality

Remember to delete all the resources you created on AWS once you’re done so as not to incur any costs!!!

GameTracker API System

AdesegunA — Mon, 03 Feb 2025 12:19:08 +0000

Project Overview

This project demonstrates building a robust and scalable API management system for querying real-time sports data. It leverages the power of Amazon's cloud services, including:

Amazon ECS (Fargate): for containerized, serverless execution of the backend application.
Amazon API Gateway: for exposing secure and scalable RESTful endpoints for querying sports data.
Amazon CloudWatch: for monitoring and logging application performance and health. This project showcases best practices in cloud architecture, including:
API Management: API Gateway provides a central hub for managing and securing API access.
Container Orchestration: ECS Fargate simplifies container deployment and management.
Secure AWS Integrations: IAM policies ensure least privilege access for API Gateway and ECS tasks.

Features

Exposes a well-documented and easy-to-use REST API for retrieving real-time sports data.
Runs a containerized backend application using Amazon ECS with Fargate for scalability and serverless operation.
Employs a scalable and serverless architecture to handle fluctuating workloads efficiently.
Leverages Amazon API Gateway for API management and routing, simplifying API development and deployment.

Prerequisites

Sports API Key: Sign up for a free account and subscription & obtain your API Key at serpapi.com
AWS Account: Create an AWS Account & have basic understanding of ECS, API Gateway, Docker & Python
AWS CLI Installed and Configured: Install and configure the AWS CLI to interact with AWS services programmatically. You can find detailed instructions in the AWS documentation.
Serpapi Library: Install Serpapi library in local environment "pip install google-search-results"
Docker CLI and Desktop Installed: Install Docker for building and pushing container images.

Technical Architecture

Technologies

Cloud Provider: AWS
Core Services: Amazon ECS (Fargate), API Gateway, CloudWatch
Programming Language: Python 3.x
Containerization: Docker
IAM Security: Custom least privilege policies for ECS task execution and API Gateway

Project Structure

sports-api-management/
├── app.py # Flask application for querying sports data
├── Dockerfile # Dockerfile to containerize the Flask app
├── requirements.txt # Python dependencies
├── .gitignore
└── README.md # Project documentation

Setup Instructions

Clone the Repository

git clone https://github.com/adesegunasunmo/sports-api-container.git
cd containerized-sports-api

Create ECR Repo

aws ecr create-repository --repository-name sports-api --region us-east-1

Authenticate Build and Push the Docker Image

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <AWS_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com

docker build --platform linux/amd64 -t sports-api .
docker tag sports-api:latest <AWS_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/sports-api:sports-api-latest
docker push <AWS_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/sports-api:sports-api-latest

Set Up ECS Cluster with Fargate

Create an ECS Cluster:
Go to the ECS Console → Clusters → Create Cluster
Name your Cluster (sports-api-cluster)
For Infrastructure, select Fargate, then create Cluster

Create a Task Definition:
Go to Task Definitions → Create New Task Definition
Name your task definition (sports-api-task)
For Infrastructure, select Fargate
Add the container:
- Name your container (sports-api-container)
- Image URI: .dkr.ecr.us-east-1.amazonaws.com/sports-api:sports-api-latest
- Container Port: 8080
- Protocol: TCP
- Port Name: Leave Blank
- App Protocol: HTTP
Define Environment Eariables:
- Key: SPORTS_API_KEY
- Value:
- Create task definition

Run the Service with an ALB
Go to Clusters → Select Cluster → Service → Create.
Capacity provider: Fargate
Select Deployment configuration family (sports-api-task)
Name your service (sports-api-service)
Desired tasks: 2
Networking: Create new security group
Networking Configuration:
- Type: All TCP
- Source: Anywhere
Load Balancing: Select Application Load Balancer (ALB).
ALB Configuration:
- Create a new ALB:
- Name: sports-api-alb
- Target Group health check path: "/sports"
- Create service

Test the ALB:
After deploying the ECS service, note the DNS name of the ALB (e.g., sports-api-alb-.us-east-1.elb.amazonaws.com)
Confirm the API is accessible by visiting the ALB DNS name in your browser and adding /sports at end (e.g, http://sports-api-alb-.us-east-1.elb.amazonaws.com/sports)

Configure API Gateway

Create a New REST API:
Go to API Gateway Console → Create API → REST API
Name the API (e.g., Sports API Gateway)
Set Up Integration:
Create a resource /sports
Create a GET method
Choose HTTP Proxy as the integration type
Enter the DNS name of the ALB that includes "/sports" (e.g. http://sports-api-alb-.us-east-1.elb.amazonaws.com/sports
Deploy the API:
Deploy the API to a stage (e.g., prod)
Note the endpoint URL

Test the System

Use curl or a browser to test:

curl https://<api-gateway-id>.execute-api.us-east-1.amazonaws.com/prod/sports

What We Learned

Deploying a Scalable, Containerized Application with Amazon ECS and exposing Public APIs via API Gateway.

Future Enhancements

Add caching for frequent API requests using Amazon ElastiCache
Add DynamoDB to store user-specific queries and preferences
Secure the API Gateway using an API key or IAM-based authentication
Implement CI/CD for automating container deployments rmation layer with AWS Glue ETL.
Add advanced analytics and visualizations using AWS QuickSight.

Important Reminder:
Don't forget to delete all the resources you created once you're done. Services like S3, ECS, ECR e.t.c, can incur costs, so make sure to clean up!
Thank you for reading, and I hope you found this guide helpful. Keep learning and building!

NBA Game Day Notifications /Sports Alerts System

AdesegunA — Tue, 21 Jan 2025 13:37:52 +0000

Project Overview

Stay on top of the game!This project is a real time NBA Game Day Notifications System that delivers live score updates straight to your inbox or phone via SMS/Email. Built using AWS Services and NBA APIs, it showcases how to design a secure, scalable, and automated cloud-based notification system. Perfect for sports enthusiasts and techies exploring AWS and Python-based solutions.

Key Features

Live Score Updates: Fetches live NBA scores from the NBA Game API.
Real-Time Notifications: Sends formatted updates via Amazon SNS (SMS/Email).
Automated Scheduling: Uses Amazon EventBridge for regular updates.
Security-Focused Design: Implements the principle of least privilege for AWS IAM roles.

Tech Stack

Cloud Provider: AWS
Core Services: Amazon SNS, AWS Lambda, EventBridge
API: NBA Game API from SportsData.io
Language: Python 3.x
Security: Custom IAM policies for secure integration

Architecture Overview

Amazon EventBridge: Automates scheduling of the Lambda function to fetch live game data.
AWS Lambda: Executes the notification logic in real time.
Amazon SNS: Sends game day updates to users via SMS and Email.

Project Structure

game-day-notifications/
├── src/
│   ├── gd_notifications.py          # Main Lambda function code
├── policies/
│   ├── gd_sns_policy.json           # SNS publishing permissions
│   ├── gd_eventbridge_policy.json   # EventBridge to Lambda permissions
│   └── gd_lambda_policy.json        # Lambda execution role permissions
├── .gitignore
└── README.md                        # Project documentation

Getting Started

Prerequisites
- NBA API Key: Get your free API key from SportsData.io
- AWS Account: Familiarity with AWS services and Python is a plus!
Clone the Repository

git clone git@github.com:adesegunasunmo/Game-Day-Notifications.git
cd Game-Day-Notifications

Step-by-Step Setup

Create an SNS Topic

Open the AWS Management Console.
Navigate to the SNS service.
Click Create Topic and select Standard as the topic type.
Name the topic (e.g., gd_topic) and note the ARN.
Click Create Topic.

Add Subscriptions to the SNS Topic

After creating the topic, click on the topic name from the list.
Navigate to the Subscriptions tab and click Create subscription.
Select a Protocol: For Email:
- Choose Email.
- Enter a valid email address.
- For SMS (phone number):
- Choose SMS.
- Enter a valid phone number in international format (e.g., +1234567890).
Click Create Subscription.
If you added an Email subscription:
Check the inbox of the provided email address.
Confirm the subscription by clicking the confirmation link in the email.
For SMS, the subscription will be immediately active after creation.

Create an SNS Topic Policy

Open the IAM service in the AWS Management Console.
Navigate to Policies → Create Policy.
Click JSON and paste the JSON policy from gd_sns_policy.json file
Replace REGION and ACCOUNT_ID with your AWS region and account ID.
Click Next: Tags (you can skip adding tags).
Click Next: Review.
Enter a name for the policy (e.g., gd_sns_policy).
Review and click Create Policy.

Create an IAM Role for Lambda

Open the IAM service in the AWS Management Console.
Click Roles → Create Role.
Select AWS Service and choose Lambda.
Attach the following policies:
SNS Publish Policy (gd_sns_policy) (created in the previous step).
Lambda Basic Execution Role (AWSLambdaBasicExecutionRole) (an AWS managed policy).
Click Next: Tags (you can skip adding tags).
Click Next: Review.
Enter a name for the role (e.g., gd_role).
Review and click Create Role.
Copy and save the ARN of the role for use in the Lambda function.

Deploy the Lambda Function

Open the AWS Management Console and navigate to the Lambda service.
Click Create Function.
Select Author from Scratch.
Enter a function name (e.g., gd_notifications).
Choose Python 3.x as the runtime.
Assign the IAM role created earlier (gd_role) to the function.
Under the Function Code section:
Copy the content of the src/gd_notifications.py file from the repository.
Paste it into the inline code editor.
Under the Environment Variables section, add the following:
NBA_API_KEY: your NBA API key.
SNS_TOPIC_ARN: the ARN of the SNS topic created earlier.
Click Create Function.

Automate with Eventbridge

Navigate to the Eventbridge service in the AWS Management Console.
Go to Rules → Create Rule.
Select Event Source: Schedule.
Set the cron schedule for when you want updates (e.g., hourly).
Under Targets, select the Lambda function (gd_notifications) and save the rule.

Testing the System

Open the Lambda function in the AWS Management Console.
Create a test event to simulate execution.
Run the function and check CloudWatch Logs for errors.
Verify that SMS notifications are sent to the subscribed users.

What I Learned

Setting up a cloud-based notification system with AWS SNS and Lambda.
Best practices for securing AWS resources with IAM..
Integrating third-party APIs into serverless workflows.
Automating tasks with EventBridge and Lambda.

Remember to delete all the resources you created on AWS once you’re done so as not to incur any costs!

Thank you for reading, and I hope you found this guide helpful. Keep learning and building!

Building a Weather Data Collection System with AWS S3 and OpenWeather API

AdesegunA — Tue, 21 Jan 2025 12:13:38 +0000

Project Overview

This project is a Weather Data Collection System that demonstrates core DevOps principles by combining:

External API Integration (OpenWeather API)
Cloud Storage (AWS S3)
Infrastructure as Code
Version Control (Git)
Python Development
Error Handling
Environment Management

Features

Fetches real-time weather data for multiple cities
Displays temperature (°F), humidity, and weather conditions
Automatically stores weather data in AWS S3
Supports multiple cities tracking
Timestamps all data for historical tracking

Technical Architecture

Language: Python 3.x
Cloud Provider: AWS (S3)
External API: OpenWeather API
Dependencies:
- boto3 (AWS SDK)
- python-dotenv
- requests

Architectural Review

Project Structure

weather-dashboard/
  src/
    __init__.py
    weather_dashboard.py
  tests/
  data/
  .env
  .gitignore
  requirements.txt

Prerequisites

Ensure the following tools and credentials are set up on your local machine before proceeding:

VS Code Editor you can see documentation here
- AWS CLI you can see documentation here
- AWS Account: Access credentials with sufficient permissions to interact with S3.
- OpenWeather API Key: Generate an API key by signing up at OpenWeather API.

Setup Instructions

Clone the repository:

  git clone git@github.com:adesegunasunmo/30-Days-DevOps-Challenge.git

Create a Virtual Environment

Before proceeding with the installation, it’s highly recommended to create a virtual environment for Python. This ensures that your package dependencies are isolated from the global environment. The installation might fail if the package is managed externally (e.g., through apt). By isolating the environment, you can safely install and manage the required dependencies without interference.

Then you have to activate the virtual environment

source envname/bin/activate

To do this:

Create a virtual environment

    python3 -m venv (your virtual environment name)

Install dependencies

   pip install -r requirements.txt

Configure environment variables (.env):

CopyOPENWEATHER_API_KEY=your_api_key
AWS_BUCKET_NAME=your_bucket_name

4.Configure AWS credentials:

aws configure

5.Run the Application Execute the Python script to fetch and upload weather data:

   python src/weather_dashboard.py

What We Learned

Working on this project helped me gain hands-on experience with:

AWS S3: Creating and managing buckets.
Environment Variable Management: Securing sensitive API keys.
Python Best Practices: Writing clean, modular, and reusable code for API integration.
Git Workflow: Using version control effectively for project development.
Error Handling: Managing failures in distributed systems gracefully.
Cloud Resource Management: Leveraging cloud services efficiently.

Future Enhancements

Add weather forecasting functionality.
Implement data visualization for weather trends.
Expand the city tracking feature.
Create automated testing for better reliability.
Set up a CI/CD pipeline for streamlined deployments.

Congratulations!

We have successfully completed "Building a Weather Data Collection System with AWS S3 and OpenWeather API!" . Great job!

I know it took a lot of clicking and configurations to set everything up. While tool like Docker can automate this process, going through it manually is a great way to understand how all the components work together.

Important Reminder:
Don't forget to delete all the resources you created once you're done. Services like S3 e.t.c, can incur small costs, so make sure to clean up!
Thank you for reading, and I hope you found this guide helpful. Keep learning and building!