DEV Community: Gbenga Ojo-Samuel

Full-Stack Serverless Inventory Management System with AWS Lambda, API Gateway, DynamoDB, and React Frontend.

Gbenga Ojo-Samuel — Mon, 21 Apr 2025 18:15:35 +0000

This project demonstrates how to build a clean and scalable full-stack inventory management system using pure AWS serverless components. With React on the frontend and Terraform on the backend, it’s cloud-native and production-ready.

📁 Project Architecture

This project is a fully serverless full-stack inventory management system built using:

AWS Lambda for serverless backend logic
Amazon API Gateway for HTTP endpoints
Amazon DynamoDB as the NoSQL database
React.js for the frontend
Terraform for infrastructure as code

🔧 Tech Stack

Frontend: React.js (calls API Gateway endpoints)
Backend: AWS Lambda (Python) + API Gateway
Database: Amazon DynamoDB
Infrastructure: Terraform
API Testing: Postman

Frontend Implementation

The React frontend provides an intuitive interface for inventory operations.The frontend communicates with backend services via API calls.

// src/App.jsx
import Header from './components/Header';
import InventoryList from './components/InventoryList';
import InventoryForm from './components/InventoryForm';
import InventoryDetails from './components/InventoryDetails';
import { fetchItems, fetchItem, createItem, updateItem, deleteItem } from './services/api';
import './App.css';

function App() {
  const [items, setItems] = useState([]);
  const [selectedItem, setSelectedItem] = useState(null);
  const [isLoading, setIsLoading] = useState(false);
  const [error, setError] = useState(null);
  const [isFormOpen, setIsFormOpen] = useState(false);
  const [isEditing, setIsEditing] = useState(false);

  // Fetch all inventory items on component mount
  useEffect(() => {
    fetchInventoryItems();
  }, []);

  const fetchInventoryItems = async () => {
    setIsLoading(true);
    setError(null);
    try {
      const data = await fetchItems();
      setItems(data.items || []);
    } catch (err) {
      setError('Failed to fetch inventory items. Please try again.');
      console.error(err);
    } finally {
      setIsLoading(false);
    }
  };

Backend Services

Lambda functions handle the core business logic of the application.

✅ CRUD Lambda Functions (Python)

create_item.py

import json
import boto3
import os
import uuid
from datetime import datetime
from decimal import Decimal

def lambda_handler(event, context):
    # Initialize DynamoDB
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table(os.environ.get('DYNAMODB_TABLE'))

    try:
        # Parse request body
        body = json.loads(event.get('body', '{}'), parse_float=Decimal)  # Convert floats to Decimal

        # Generate unique itemId and timestamps
        item_id = str(uuid.uuid4())
        timestamp = datetime.utcnow().isoformat()

        # Create item object
        item = {
            'itemId': item_id,
            'name': body.get('name'),
            'description': body.get('description'),
            'quantity': int(body.get('quantity', 0)),  # Ensure quantity is integer
            'price': Decimal(str(body.get('price', 0))),  # Convert price to Decimal
            'category': body.get('category'),
            'imageUrl': body.get('imageUrl'),
            'createdAt': timestamp,
            'updatedAt': timestamp
        }

Other functions follow a similar pattern for GET, PUT, and DELETE using event['pathParameters'] and event['body'].

API Gateway Routes
These routes define the RESTful endpoints for the Inventory Management System. Each route is connected to an AWS Lambda function:

Method	Path	Lambda
POST	/items	create_item.py
GET	/items	get_items.py
PUT	/items/{id}	update_item.py
DELETE	/items/{id}	delete_item.py

DynamoDB Data Model
DynamoDB provides a flexible, high-performance database solution. Access patterns are optimized through careful index design to support common inventory operations.

{
  "itemId": "item-123",
  "name": "Wireless Headphones",
  "description": "WH-2023-BLK",
  "category": "electronics",
  "quantity": 157,
  "imageUrl": "supplier-456",
  "price": 45.99,
  "createdAt": "timestamp",
  "updatedAt": "timestamp"

Deploying the Backend with Terraform

Terraform enables infrastructure as code, making deployment consistent and repeatable. Here's a step-by-step guide to deploying the backend infrastructure:

1. Prerequisites
Before starting, ensure you have:

AWS CLI installed and configured with appropriate permissions
Terraform CLI installed
Git installed for repository access

2. Clone the Repository
First, clone the project repository containing the Terraform configuration:

git clone https://github.com/OjoOluwagbenga700/serverless-inventory-management-system.git
cd backend/terraform

3. Initialize Terraform

Initialize the Terraform working directory:

terraform init

This command downloads the necessary provider plugins and sets up the backend configuration.

4. Review the Execution Plan
Generate and review the execution plan:

terraform plan

This shows you what resources will be created, modified, or destroyed.

5. Apply the Configuration
Apply the Terraform configuration to provision the infrastructure:

terraform apply --auto-approve

Terraform will create all the necessary AWS resources:

DynamoDB tables
IAM roles and policies
Lambda functions
API Gateway endpoints
CloudWatch log groups

6. Verify Deployment
After successful application, Terraform will output the API Gateway endpoint URL

api_gateway_url = "https://i5mmj6bhw1.execute-api.us-east-1.amazonaws.com"

7. Update Frontend Configuration
Use the output values to update your frontend environment configuration.

// src/services/api.js
// API Gateway integration layer

const API_ENDPOINT = 'https://i5mmj6bhw1.execute-api.us-east-1.amazonaws.com';

// Fetch all inventory items
export async function fetchItems() {
  try {
    const response = await fetch(`${API_ENDPOINT}/items`);

    if (!response.ok) {
      const errorData = await response.json();
      throw new Error(errorData.message || `API error: ${response.status}`);
    }

    return await response.json();
  } catch (error) {
    console.error('Error fetching items:', error);
    throw error;
  }
}

7. Run frontend Application

Navigate to the frontend directory

cd frontend/inventory-management-system

Run the following npm commands

npm install
npm start

Testing CRUD Operations with Postman

You can test all CRUD operations by:

Setting method and endpoint (e.g., POST /items)
Sending appropriate JSON body
Checking responses

Conclusion
By leveraging AWS serverless technologies, React, and Terraform, businesses can implement a powerful inventory management system that scales with their needs while minimizing operational overhead. This cloud-native approach provides the flexibility to adapt quickly to changing business requirements while maintaining robust security and performance.
Feel free to clone, fork, or build on top of it!

Containerized Sports API for Premier League Game Day Schedule with AWS

Gbenga Ojo-Samuel — Thu, 20 Feb 2025 13:04:45 +0000

Introduction

Sports enthusiasts and developers often require real-time game schedules for various leagues. This article demonstrates how to build a containerized API using AWS Elastic Container Registry (ECR), Elastic Container Service (ECS), Application Load Balancer (ALB), and API Gateway, with an ECS-hosted Flask application that retrieves Premier League game schedules from SerpAPI. The entire setup will be provisioned using Terraform.

Architecture Overview

Flask Application: A Python-based application that fetches EPL game schedules from SerpAPI.

ECR: Stores the Docker image of the Flask application.

ECS (Fargate): Runs the containerized application in a scalable environment.

ALB: Routes traffic to the ECS service.

API Gateway: Acts as an entry point for external requests.

Prerequisites

AWS Account with necessary IAM permissions
Terraform installed on the local machine
AWS CLI configured with appropriate credentials
Docker installed for containerization
SerpAPI key for fetching Premier League schedules

Tech Stack

Programming Language: Python (Flask framework)
Containerization: Docker
Infrastructure as Code: Terraform
Cloud Provider: AWS (ECR, ECS Fargate, ALB, API Gateway)
External API: SerpAPI

Implementation Steps

Flask Application

The Flask application in app.py fetches Premier League schedules from SerpApi

Provision AWS Resources Using Terraform

The full terraform codes can be found in the repository below

https://github.com/OjoOluwagbenga700/containerized_sport_api.git

Create custom VPC and Security Groups

Create ECR, build Docker image and push image to ECR

Dockerfile

Create ECS Cluster,ECS Role/Permissions, ECS Task Definitions and ECS Service

Create Application Load Balancer and Target Group

Create API Gateway

Deploy and Test

Run the following commands to provision the infrastructure:

terraform init
terraform plan 
terraform apply -auto-approve

Conclusion

This setup provides a fully automated, scalable, containerized API for retrieving Premier League schedules in real-time. By leveraging Terraform, ECS, ECR, ALB, and API Gateway, the architecture ensures reliability, security, and efficiency for sports API solutions.

To Clean up

Run the command below

terraform destroy --auto-approve

Sports Data Backup - DynamoDB, ECS, AWS MediaConvert & EventBridge

Gbenga Ojo-Samuel — Wed, 12 Feb 2025 15:07:45 +0000

Introduction
Managing and processing sports data efficiently requires a robust and scalable infrastructure. This project integrates Amazon DynamoDB to store game data fetched from RapidAPI, ensuring persistent storage, scalability, and efficient querying for processing and analytics. Additionally, we leverage AWS Systems Manager (SSM) Parameter Store to securely manage API keys, enhancing security and maintainability. Furthermore, AWS MediaConvert is utilized for transcoding game highlight videos stored in S3 Bucket, ensuring optimal video quality and compatibility across multiple platforms.

Solution Architecture

RapidAPI fetches NCAA game data:
• stores the raw json sports data in s3 bucket
• stores structured sports data for quick retrieval and long-term storage in Amazon DynamoDB.
Containerized Processing with ECS:
• ECS (Fargate) hosts a containerized service that processes game data.
• retrieves secured API keys from AWS Systems Manager Parameter Store
• AWS ECR stores and manages Docker images
Video Transcoding with AWS MediaConvert:
• video clips quality is enhanced and the output save back in S3 using AWS MediaConvert.
Monitoring and Alerting with CloudWatch:
• cloudWatch Logs capture application and infrastructure logs.
• cloudWatch Alarms notify on failures or performance degradation.

Prerequisites

Before we get started, ensure you have the following:

RapidAPI Account: Sign up for a RapidAPI account and subscribe to an API that provides NCAA game highlights.
AWS Account: AWS access with the required permission to access the necessary services.
Docker Installed: Install Docker on your system to run the containerized workflow.
Terraform Installed: Ensure Terraform is installed for infrastructure deployment.
Basic CLI Knowledge: Familiarity with using command-line tools for API requests, AWS configurations, and Terraform commands.

Tech Stack

Python
AWS ECR, ECS, DynamoDB, & Elemental MediaConvert
Docker
Terraform

Step 1: Clone the project Repository

Use the git clone command to clone the project repository to your local machine.

https://github.com/OjoOluwagbenga700/sports-data-backup.git

Ensure that you have Git installed and configured on your system.

Navigate to Repository: cd sports-data-backup

Step 2: Reviewing the content of the code

let's take a closer look at the file structure and its contents.

app folder: Contains main application configuration files, with each Python script serving a specific function.

config.py: Loads environment variables, assigning defaults where needed.
fetch.py: Retrieves highlights from the API and stores them as a JSON file in S3 and also as structured data in DynamoDB
process_videos.py: Extracts videos URLs from JSON data file in S3, download them , and save them to S3 under the videos/ folder. Logs each step.
mediaconvert_process.py: Processes videos using MediaConvert, configuring codec, resolution, bitrate, and audio settings before storing them back in S3.
run_all.py: Executes scripts sequentially, ensuring proper task execution with buffer time.
Dockerfile: Provides the step by step approach to build the docker image.

container_definitions.tpl: A template file for defining ECS container definitions in Terraform.

Terraform Infrastructure Files: These .tf files define the AWS infrastructure using Terraform.

dynamodb.tf: Configures the Amazon DynamoDB table to store sports data.
ecr.tf: Defines the AWS Elastic Container Registry (ECR) for storing Docker images.
ecs.tf: Configures AWS Elastic Container Service (ECS) to run the application.
iam.tf: Defines IAM roles and permissions required for AWS services.
networking.tf: Sets up networking components (VPC, subnets, security groups).
provider.tf: Specifies the AWS provider configuration for Terraform.
s3.tf: Manages S3 buckets for storing raw and processed videos.
scheduler.tf: Defines scheduled tasks (potentially using CloudWatch events or EventBridge scheduler).
ssm.tf: Configures AWS Systems Manager (SSM) Parameter Store for secret management.
terraform.tfvars: Contains variable values used in Terraform.
variable.tf: Declares input variables for Terraform configurations.

Step 3: Running Terraform Commands

Initialize Terraform: Run terraform initto initialize the Terraform working directory and download the necessary plugins.

Plan the Deployment: Run terraform planto preview the resources that Terraform will create.

Apply the Configuration: Run terraform apply –auto-approve to deploy the infrastructure.

Step 4: Verify Application deployment by confirming resources deployed on AWS

SSM Parameter Store

EventBridge Scheduler

S3 Bucket

AWS ECR

AWS ECS

AWS Elemental MediaConvert

DynamoDB

AWS Cloudwatch Logs

Conclusion

This solution enhances reliability by fetching data from RapidAPI and storing it in Amazon DynamoDB, improving processing with ECS, refining video workflows using MediaConvert, and adding robust monitoring with CloudWatch. Additionally, AWS SSM Parameter Store ensures secure management of API keys and secrets, reducing exposure risks and improving security. By using Terraform, we ensure a repeatable and scalable infrastructure.

Game Highlights Processor with AWS, Docker & Terraform.

Gbenga Ojo-Samuel — Fri, 07 Feb 2025 12:35:41 +0000

This project builds on our previous NCAA game highlight processing pipeline (link below), which used a deployment script. We're now implementing a full end-to-end deployment using Terraform. This Infrastructure-as-Code approach allows us to manage AWS resources consistently and scalably, automating video ingestion, processing, and delivery.

https://dev.to/gbenga700/streamlining-ncaa-game-highlights-with-rapidapi-docker-and-aws-mediaconvert-22li

Technical Diagram

RapidAPI is a robust API marketplace offering access to a variety of data sources, including NCAA game highlights. By integrating RapidAPI into our workflow, we can easily retrieve game highlight videos and relevant metadata.

AWS ECR provides a secure, scalable container registry for storing and managing Docker images while AWS ECS enables seamless deployment and orchestration of these containers.
Together, they ensure workflow portability and reproducibility across environments by encapsulating dependencies and application code within Docker images.

AWS MediaConvert is a powerful tool for converting media files into formats optimized for various use cases, such as streaming or archival. In this project, MediaConvert processes the raw video fetched from RapidAPI and converts it into desired output formats

Prerequisites
Before we get started, ensure you have the following:

RapidAPI Account: Sign up for a RapidAPI account and subscribe to an API that provides NCAA game highlights.
AWS Account: AWS access with the required permission to access the necessary services.
Docker Installed: Install Docker on your system to run the containerized workflow.
Terraform Installed: Ensure Terraform is installed for infrastructure deployment.
Basic CLI Knowledge: Familiarity with using command-line tools for API requests, AWS configurations, and Terraform commands.

Tech Stack

Python
AWS ECR, ECS & Elemental MediaConvert
Docker
Terraform

Step 1: Clone the project Repository

By cloning the project Repository, we'll have access to the application codes and terraform configurations needed for our deployment process.

Clone Repository: Use the

git clone

command to clone the Terraform code repository to your local machine. Ensure that you have Git installed and configured on your system.

https://github.com/OjoOluwagbenga700/game_highlight_processor_terraform.git

Navigate to Repository: cd game_highlight_processor_terraform

Step 2: Reviewing the content of the code

Now that we've cloned the project code repository, let's take a closer look at the file structure and its contents

app folder: This folder contains the main application configuration files. Each python script has a specific function to perform within the application.

config.py: Imports necessary environment variables and assigns them to Python variables, providing default values where appropriate.

fetch.py: fetches the highlights from the API and store them in an S3 bucket as a JSON file.

process_one_video.py: Connects to the S3 bucket and retrieves the JSON file. Extracts the first video URL from within the JSON file. Downloads the video file from the internet into the memory using the requests library. Saves the video as a new file in the S3 bucket under a different folder (videos/) and logs the status of each step.

mediaconvert_process.py: Uses MediaConvert to process a video file - configures the video codec, resolution and bitrate. Also configured the audio settings and stores the processed video back into S3 bucket.

run_all.py: Runs the scripts in a chronological order and provides buffer time for the tasks to be created.

Dockerfile: Provides the step by step approach to build the docker image.

ecr.tf: This terraform code will create:
• private repository on Amazon ECR
• build Docker image from the app folder.
• push the image to the repository.

ecs.tf: This terraform code will create:
• an ecs cluster
• a task definition
• an ecs service
• an ecs cloudwatch log group

iam.tf: This terraform code will create the necessary permissions and policies for AWS ECR, ECS , S3 Bucket and AWS Elemental Mediaconvert.

s3.tf: This terraform code will create the s3 bucket for storing our fetched json data and the processed videos.

networking.tf: This terraform code will create:
• custom vpc, internet gateway, subnets
• security group for the ecs task

Step 3: Running Terraform Commands
Initialize Terraform: Run terraform init to initialize the Terraform working directory and download the necessary plugins.

Plan the Deployment: Run terraform plan to preview the resources that Terraform will create.

Apply the Configuration: Run terraform apply –auto-approve to deploy the infrastructure.

Step 4: Verify Application deployment by confirming resources deployed on AWS

S3 Bucket

AWS ECR

AWS ECS

AWS Elemental MediaConvert

AWS Cloudwatch Logs

Conclusion
Automating NCAA game highlights with RapidAPI, AWS ECS, AWS ECR, AWS MediaConvert, and Terraform provides a robust and scalable solution for processing sports footage efficiently. This approach eliminates manual overhead, ensures consistent deployments, and optimizes video delivery for various platforms.

Thanks for Reading!!!

Streamlining NCAA Game Highlights with RapidAPI, Docker, and AWS MediaConvert

Gbenga Ojo-Samuel — Tue, 28 Jan 2025 10:33:53 +0000

In this project, we demonstrate how to leverage RapidAPI, Docker, and AWS MediaConvert to create an efficient pipeline for obtaining and processing NCAA game highlights. Additionally, we use Terraform to deploy an EC2 instance that automatically establishes an SSH connection and runs Python scripts to manage the workflow seamlessly. This setup simplifies the process of acquiring high-quality sports media content and converting it into usable formats for distribution or analysis.

Prerequisites
Before you get started, ensure you have the following:

RapidAPI Account: Sign up for a RapidAPI account and subscribe to an API that provides NCAA game highlights.
AWS Account: Set up Role with access to AWS MediaConvert and an S3 bucket.
Docker Installed: Install Docker on your system to run the containerized workflow. We will install Docker on the EC2 instance
Terraform Installed: Ensure Terraform is installed for infrastructure deployment.
Basic CLI Knowledge: Familiarity with using command-line tools for API requests, AWS configurations, and Terraform commands.

Technical Diagram

Docker containers ensure the workflow is portable and reproducible across environments. By encapsulating the necessary dependencies and scripts in a Docker image.

Deploying an EC2 Instance with Terraform
Terraform automates the deployment of an EC2 instance that serves as the base for running the Python scripts and Dockerized workflow. This step involves:

Terraform Configuration:

Create a Terraform script that includes the following:
An EC2 instance with appropriate IAM roles and security groups to enable necessary access.
Configuration using the null_resource provisioner to upload the required Python scripts to the EC2 instance and execute them automatically.

Initialize Terraform
Clone the git repo and cd to the project directory

https://github.com/OjoOluwagbenga700/NCAAGameHighlight.git

Run terraform init to initialize the Terraform working directory.

Plan the Deployment

Use terraform plan to preview the resources that Terraform will create.

Apply the Configuration

Execute terraform apply –auto-approve to deploy the infrastructure.

Automated Script Execution

The EC2 instance is configured using the null_resource provisioner in Terraform to both upload the necessary Python scripts and execute them automatically.

The remote-exec provisioner ensures that the EC2 instance is prepared by installing Docker, setting up the environment, and running the highlight-fetching workflow in a containerized environment.

*Verify Deployment on AWS *

Conclusion

Congratulations! We have successfully demonstrated how to fetch JSON data from RapidAPI, transform it into a video, and enhance its quality using AWS MediaConvert.

This project underscores the power of integrating APIs, Docker containers, and AWS cloud tools to build a streamlined and automated workflow. By leveraging Terraform for infrastructure as code, we ensured consistent deployment of an EC2 instance to automate the entire process, from fetching NCAA game highlights to enhancing them for better quality.

Serverless NBA Data Lake Application with API Gateway, AWS Lambda, Amazon S3, AWS Glue and Athena Using Terraform

Gbenga Ojo-Samuel — Wed, 15 Jan 2025 13:57:18 +0000

In sports analytics, the ability to process and analyze vast amounts of data in real time has become a game-changer. Having the power to ingest, store, and query large datasets of NBA statistics seamlessly and also enjoying the scalability and cost-efficiency of serverless architecture is awesome.
In this project, we’ll explore how to build a Serverless NBA Data Lake Application using API Gateway, AWS Lambda, Amazon S3, AWS Glue, and Amazon Athena — all orchestrated with Terraform.

System Architecture Overview
The architecture leverages the following components:
• Amazon S3: Serves as the central data lake for storing raw, processed, and curated NBA data in JSON format.
• AWS Lambda: Lambda functions Fetches NBA Data from sportdata.io, formats it and upload to Amazon S3
• Amazon API Gateway: Provides a RESTful API that triggers the Lambda function to fetch NBA data from sportdata.io and upload to an S3 bucket.

• AWS Glue: Automatically discovers and catalogs the data stored in S3 into a schema using the Glue Database Catalog and Glue crawler for efficient querying.
• Amazon Athena: Enables serverless querying of the data lake using standard SQL, allowing users to retrieve insights from the curated NBA data and store result in an Amazon S3 bucket

Prerequisites:
• AWS account with required access and permission to configure services such as Lambda, S3, Glue API Gateway and Athena.
• Experience with programming languages supported by AWS Lambda, such as Python.
• Terraform installed on your local machine
• AWS CLI Installed and configured on your local machine.

Define Your Lambda function
We will develop a Python script for our Lambda function to retrieve NBA data from sportdata.io, process it, and uploads it to Amazon S3. The complete python code is available in the repository.

Terraform Configuration
We will use Terraform modules for this deployment to ensure modularity, reusability, and maintainability in our infrastructure as code. Each folder in the modules directory will define the infrastructure configurations required for deploying specific AWS services. See below

• API Gateway Module: This module deploys an API Gateway that will serve as a trigger to the lambda function to retrieve data from sportdata.io and upload it in Amazon S3.

• iam_role Module: This module contains the terraform codes that defines the necessary permissions for lambda to be able to retrieve and upload NBA data to Amazon S3 and API Gateway to be able to trigger the lambda function.

• Lambda Module: This module defines the terraform codes for archiving the python code in a zip file and also create a lambda function that retrieves NBA data from sportsdata.io, process it and uploads to Amazon S3.

• S3 module: This module defines the terraform codes that creates the Amazon S3 bucket that will be used to store data retrieved form sportdata.io by the lambda function.

• glue module: This module defines the terraform codes that creates the Amazon Glue catalogs database, Glue crawler and Glue table which automatically discovers the data stored in S3 and catalogs it into a schema for efficient querying.

• athena module: This module defines the terraform codes that creates an Athena workgroup that enables serverless querying of the sport data lake stored in S3 using standard SQL.

Check the link below for the full terraform configurations

https://github.com/OjoOluwagbenga700/sport-data-lake.git

Step 1: Clone the Terraform Code
By cloning the Terraform code, we'll have access to the infrastructure-as-code configurations needed for our deployment process.

Clone Repository: Use the git clone command to clone the Terraform code repository to your local machine.

Ensure that you have Git installed and configured on your system.

https://github.com/OjoOluwagbenga700/sport-data-lake.git

Change directory to the folder name sport-data-lake.

Ensure you update the terraform.tfvars file with your API Key from sportdata.io

Step 2: Running Terraform Commands

Terraform init: Initialize Terraform in the project directory to download necessary plugins and modules.

Terraform Plan: Generate an execution plan to preview the changes that Terraform will make to the infrastructure.

Terraform Apply: Run terraform apply --auto-approve to deploy the infrastructure on AWS.

Step 3: Confirm resources deployed on AWS

Lambda Function

Glue crawler

Glue catalog database and Table

S3 Bucket without Data upload

Athena Workgroup

API Gateway

Step 4: Testing the Application

To trigger the lambda function to retrieve, process and upload NBA data to S3, we will send a GET request through the API Gateway Invoke URL.

Copy the API Gateway invoke url to your browser, add /dev/data to indicate the API stage and path and click enter.

https://r3zks22udh.execute-api.us-east-1.amazonaws.com/dev/data

NBA Data Uploaded into S3

Preview data table in Athena

Performing Simple SQL query in Athena

Athena Query Result

Query results are stored in a defined folder in the s3 bucket and can be downloaded accordingly. See below

Conclusion: Congratulations!!!, we have successfully built a Serverless NBA Data Lake Application by leveraging AWS services like API Gateway, Lambda, S3, Glue, and Athena. Terraform adds to the elegance by ensuring your infrastructure is provisioned consistently and can be replicated or modified with ease. This architecture not only showcases the potential of serverless computing but also opens up endless possibilities for expanding into other domains, such as real-time analytics, machine learning, or personalized user experiences.

To Clean up: Run terraform destroy to delete all infrastructure deployed by the terraform codes.

Game Notification System with Amazon Event Scheduler, AWS Lambda, and Amazon SNS Using Terraform

Gbenga Ojo-Samuel — Mon, 13 Jan 2025 12:46:01 +0000

In this project I will utilize Terraform to automate the provisioning of a game notification system using Amazon EventBridge, AWS Lambda and Amazon SNS. Terraforms ability to manage infrastructure as code will enhance consistency, reduce errors, and accelerate the development of a robust notification system.

System Architecture Overview

• Scheduler (Amazon EventBridge): EventBridge schedules events, such as game start times, or scores updates. These scheduled events trigger Lambda functions to retrieve data from sportdata.io (NBA API).
• AWS Lambda: Lambda functions process NBA games data fetched from sportdata.io. It formats notifications with relevant information (e.g., NBA game schedules, scores or game highlights).
• Amazon SNS: Processed notifications are sent to Amazon SNS. Amazon SNS manages the distribution of messages to multiple subscribers via email, SMS, or mobile push notifications.

Prerequisites:
• AWS account with required access and permission to configure services such as Lambda, SNS, and EventBridge.
• Experience with programming languages supported by AWS Lambda, such as Python, Node.js, or Java.
• Terraform installed on your local machine
• AWS CLI Installed and configured on your local machine.

Define Your Lambda function
I will be creating a python code that our lambda function will use to retrieve NBA games data from sportdata.io, process it and publish to Amazon SNS. See complete code in the repository.

Terraform Configuration
We will be using Terraform modules for this deployment. Modules are a key feature of Terraform that promotes modularity, reusability, and maintainability in infrastructure as code. Each folder in the modules directory will contain the infrastructures required to be deployed for each AWS service. See below.

• EventBridge Module: This EventBridge scheduler sets up recurring NBA game events that triggers the lambda function.

• Iam_Role Module: This module contains the terraform codes that defines the necessary permissions for lambda to be able to publish to Amazon SNS and EventBridge to be able to trigger the lambda function.

• SNS module: This module defines the terraform codes that manages the distribution of processed notifications from lambda to multiple subscribers via email, SMS, or mobile push notifications.

Check the link below for the full terraform configurations.
https://github.com/OjoOluwagbenga700/Game-Notification-System.git

Step 1: Clone the Terraform Code
By cloning the Terraform code, we'll have access to the infrastructure-as-code configurations needed for our deployment process.
Clone Repository: Use the git clone command to clone the Terraform code repository to your local machine. Ensure that you have Git installed and configured on your system.

https://github.com/OjoOluwagbenga700/Game-Notification-System.git

Change directory to the folder name game-notification-system.

Step 2: Running Terraform Commands

Below main.tf file defines all the infrastructure that will be deployed after running the terraform commands.

Terraform init: Initialize Terraform in the project directory to download necessary plugins and modules.

Run Terraform init

Terraform Plan: Generate an execution plan to preview the changes that Terraform will make to the infrastructure

Run Terraform Plan

Terraform Apply: Apply the Terraform execution plan to create infrastructure resources as needed. Respond with yes to confirm the execution plan.

Run terraform apply --auto-approve to deploy the infrastructure on AWS.

Confirm Resources deployed on AWS

Amazon SNS Topic

Amazon SNS Subscription

Email Notification to confirm subscription

Lambda function

Environment Variables

Scheduler (Amazon EventBridge)

Cron schedule runs every 30mins from 9pm WAT

IAM Role

Game Day Notification system updates based on the cron schedule.

Conclusion: By leveraging AWS Lambda, EventBridge, and SNS, we have implemented an NBA game-day notification system using Terraform as infrastructure as code tool. This architecture enables the delivery of email notifications for your favorite NBA games.

To Clean up: Run terraform destroy to delete all infrastructure deployed by the terraform codes.

Thanks for Reading!!!

Serverless Resume API with Terraform and GitHub Actions

Gbenga Ojo-Samuel — Fri, 12 Jul 2024 10:55:32 +0000

Introduction

In today's world, building applications that automatically adjust to changing demands is crucial. A unique way to showcase this is by deploying your resume in JSON format using serverless AWS services. This guide will walk you through deploying your JSON resume data with Terraform, leveraging AWS Lambda, DynamoDB, and API Gateway. This setup ensures your resume is always available, scales seamlessly, and can be accessed through a user-friendly API.
Before we begin, ensure you have the following:

AWS Account: An active AWS account with the necessary permissions.
Terraform: Installed on your local machine.
AWS CLI: Installed and configured on your local machine.

Step 1: Define Your JSON Resume Data
Create your resume in JSON format. You can use a JSON schema from the link below:
https://jsonresume.org/schema

Step 2: Define Your Lambda function
We will be creating a python code that our lambda function will use to retrieve the resume data from DynamoDB. See complete code in the repository.

Step 3: Create Terraform Configuration
Create a new directory for your Terraform configuration files. Inside this directory, we will create a sub-directory for our modules. Each folder in the modules directory will contain the infrastructures required to be deployed for each AWS service. See below.

Api Gateway module: This describes the detailed terraform configuration for our Api Gateway.

DynamoDB Module: This module contains the terraform codes for deploying DynamoDB and also uploading the resume JSON data on it.

Iam_Role Module: This module contains the terraform codes that defines the necessary permissions for lambda function to access DynamoDB and CloudWatch.

Lambda Module: This module defines the terraform codes for archiving the python code in a zip file and also create a lambda function from same.

The full terraform configuration codes can be found in the repository below:
https://github.com/Samuel7050/AWS_RESUME_CHALLENGE.git

Step 3: Test your codes locally
Run terraform init to initialize the terraform plugins and providers

Run terraform validate to validate the terraform codes

Run terraform plan to preview the infrastructures to be deployed. Also note that the Zip file for the lambda function will also be created in the project directory.

Run terraform apply --auto-approve to deploy the infrastructure on AWS.

Copy the** api_gateway_invoke_url** as indicated in the output above and run a test locally from VScode . Ensure you include **/path_part **of the api gateway. See below

https://susia7qsb6.execute-api.us-east-1.amazonaws.com/dev/data

Paste on a web browser

To clean up locally, run terraform destroy --auto-approve

Step 4: Configure GitHub Actions
To automate the deployment process using GitHub Actions, create a .github/workflows directory in your project root directory and add a file named deploy.yml

Step 5: Configure GitHub Secrets
In your GitHub repository, navigate to Settings > Secrets and variables and add the following secrets and variable, as shown below:

Whenever there is a push action on the main branch or a pull request to the main branch, the GitHub action is automatically triggered. See result of a push action to the main branch of the repository.

The PR from my feature branch to main branch failed as seen below

The workflow failed because of already existing policy as seen in the error code below

Lets fix the error and do another PR the to main branch again to trigger the workflow

To verify, copy the api_gateway_invoke_url/data as shown below and run on your browser. Note that /data is my api gateway path_part

https://sx60706aef.execute-api.us-east-1.amazonaws.com/dev/data

Conclusion
By following these steps, you've deployed your resume in JSON format using AWS Lambda, DynamoDB, API Gateway, and automated the process using GitHub Actions. This setup ensures your resume is highly available, scalable, and easily accessible via a REST API, with the added benefit of continuous integration and deployment. You can now share your resume API endpoint with potential employers or integrate it into your personal website.

Automating User Creation and Management with a Bash Script

Gbenga Ojo-Samuel — Tue, 02 Jul 2024 08:19:05 +0000

Introduction

Managing users on a Linux system can be a daunting task, especially in environments where you need to create multiple users, assign them to specific groups, and ensure they have secure passwords. This blog will walk you through a Bash script that automates the process of user creation, group assignment, password generation, and logging. This script is particularly useful for system administrators looking to streamline user management.

The Script

The script, named create_users.sh, reads a text file containing usernames and group names, creates the users, assigns them to the specified groups, sets up their home directories with the appropriate permissions, generates random passwords, and logs all actions.

Step-by-Step Breakdown

Here’s a detailed explanation of what the script does:

1.** Script Initialization:** The script starts by checking if an input file is provided as an argument. It sets the INPUT_FILE variable to the provided argument and defines the log file and password file paths.

#!/bin/bash

# Check if the input file is provided
if [ $# -ne 1 ]; then
    echo "Usage: $0 <input_file>"
    exit 1
fi

INPUT_FILE=$1
LOG_FILE="/var/log/user_management.log"
PASSWORD_FILE="/var/secure/user_passwords.txt"

File Existence and Directory Setup : The script checks if the input file exists. It then ensures the secure directory (/var/secure) exists, creates the log and password files, and sets appropriate permissions and ownership to ensure security.

# Check if the file exists
if [ ! -f "$INPUT_FILE" ]; then
    echo "File not found: $INPUT_FILE"
    exit 1
fi

# Ensure the secure directory exists and set permissions
sudo mkdir -p /var/secure
sudo chmod 700 /var/secure

# Initialize log and password files
sudo touch $LOG_FILE $PASSWORD_FILE
sudo chmod 600 $PASSWORD_FILE
sudo chown root:root $PASSWORD_FILE

**Password Generation Function: **This function generates a random 12-character password using openssl.

# Function to generate a random password
generate_password() {
    openssl rand -base64 12
}

4.** User Creation and Group Assignment:** This is the core of the script:

It reads each line of the input file, expecting a format of user;groups.
It checks if the user or group already exists. If not, it generates a password, creates the user,creates the group sets the password, and logs these actions.
It sets the home directory permissions to 700 to ensure only the user has access.
It assigns the user to the specified groups, logging each action.

# Read the input file line by line
while IFS=';' read -r user groups; do
    # Check if the user already exists
    if id "$user" &>/dev/null; then
        echo "User $user already exists." | sudo tee -a $LOG_FILE
    else
        # Generate a random password
        password=$(generate_password)

        # Create the user with a home directory and set the password
        sudo useradd -m -s /bin/bash "$user"
        echo "$user:$password" | sudo chpasswd

        # Log the creation and password
        echo "User $user created with home directory." | sudo tee -a $LOG_FILE
        echo "$user:$password" | sudo tee -a $PASSWORD_FILE

        # Set the permissions and ownership of the home directory
        sudo chmod 700 /home/$user
        sudo chown $user:$user /home/$user

        # Assign groups to the user
        IFS=',' read -r -a group_array <<< "$groups"
        for group in "${group_array[@]}"; do
            # Check if the group exists
            if ! getent group "$group" &>/dev/null; then
                # Create the group if it does not exist
                sudo groupadd "$group"
                echo "Group $group created." | sudo tee -a $LOG_FILE
            fi
            sudo usermod -aG "$group" "$user"
            echo "User $user added to group $group." | sudo tee -a $LOG_FILE
        done
    fi
done < "$INPUT_FILE"

echo "User creation, group assignment, and logging completed." | sudo tee -a $LOG_FILE

5. Running the Script: To run the script, save it as create_users.sh, make it executable, and execute it with the input file as an argument:

chmod +x create_users.sh
sudo ./create_users.sh <input_file>

Conclusion
This script will not only create users and assign them to groups but also create any missing groups. This ensures that all specified groups are present, and users are correctly added to them. This is my stage one project of the HNG internship program. To know more about HNG internship programs please do check the links below
https://hng.tech/internship, https://hng.tech/premium

E-Commerce Website Deployment- Containerization using Amazon ECR ,ECS and Terraform as IaC tool.

Gbenga Ojo-Samuel — Mon, 13 May 2024 13:10:19 +0000

Introduction

In my recent post, I provided a summary of our journey thus far in implementing an e-commerce website, evolving from a basic to a more advanced architecture. If you missed the post, you could find it here:

https://www.linkedin.com/posts/samuel-ojo-oluwagbenga_in-the-last-two-weeks-we-embarked-on-a-journey-activity-7190371967135338496-FQcL?utm_source=share&utm_medium=member_desktop

In this article, we will delve into the utilization of containers in our website deployment strategy. The benefits of using containers include:

Isolation: Containers provide a lightweight, isolated environment for running applications, ensuring that they operate independently without interference from other processes.
Portability: With containers, we can package our application along with its dependencies into a single unit, making it easy to deploy and run consistently across different environments, from development to production.
Scalability: Containers allow us to scale our application horizontally by spinning up multiple instances of the same container, enabling us to handle increased traffic and workload demands more effectively.
Consistency: Containers ensure consistency between development, testing, and production environments, minimizing compatibility issues and reducing the likelihood of deployment errors.

We'll leverage AWS Elastic Container Registry (ECR) to store our Docker images, Amazon Elastic Container Service (ECS) to deploy our containerized website and Terraform as our infrastructure-as-code tool.

Before diving into the utilization of containers in our website deployment strategy, it's essential to understand the prerequisites.

Docker Account: A basic understanding of Docker containerization technology is required to comprehend container deployment concepts effectively. Ensure that docker daemon is running on your local machine.
AWS Account: Access to an AWS (Amazon Web Services) account is necessary to utilize AWS Elastic Container Registry (ECR) and Amazon Elastic Container Service (ECS) for container storage and deployment.

Step 1: Clone the Terraform Code

By cloning the Terraform code, we'll have access to the infrastructure-as-code configurations needed for our deployment process.

Clone Repository: Use the git clone command to clone the Terraform code repository to your local machine. Ensure that you have Git installed and configured on your system.

https://github.com/Samuel7050/Dynamic_webserver_with_Terraform.git

Navigate to Repository: Change directory to the folder name containerized_website.

Step 2: Reviewing the Content of the Terraform Codes
Now that we've cloned the Terraform code repository, let's take a closer look at its contents to familiarize ourselves with the infrastructure-as-code configurations.

We are introducing Amazon ECR and Amazon ECS modules into the existing Terraform codes used in the previous articles.

Amazon ECR – This module will create:

private repository on Amazon ECR
an ECR policy that we enable the AWS user account to get and put images in the repository.
build Docker image from the app folder.
push the image to the repository.

Dockerfile

Docker Provider- we will be using a docker provider with the required authentication to AWS ECR

Amazon ECS – This module will create:

an ECS cluster
a task execution role and policy
a task definition
ECS service

Step 3: Running Terraform Commands
Below main.tf file defines all the infrastructure that will be deployed after running the terraform commands

Initialize: Initialize Terraform in the project directory to download necessary plugins and modules.

Run Terraform init

Plan: Generate an execution plan to preview the changes that Terraform will make to the infrastructure

Run Terraform plan

Apply: Apply the Terraform execution plan to create infrastructure resources as needed. Respond with yes to confirm the execution plan.

Run Terraform apply

Confirm Resources deployed on AWS

VPC and its dependencies

AWS ECR

AWS ECS

ACM and Route53

Update the Nameservers on your domain name registrar. Am using Namecheap as its updated as shown below

Step 4: Website Test
To confirm that our containerized website is up and running, we load our custom domain on our web browser.

To Clean up: Run terraform destroy

Thanks for Reading!!!

Jenkins Server setup using Terraform

Gbenga Ojo-Samuel — Wed, 13 Mar 2024 11:33:00 +0000

In this project, we are going to setup a Jenkin server using Terraform script. This Terraform script will spin up an EC2 instance server on AWS with the required tools to run and manage a Jenkins server.
Setting up a Jenkins server on AWS EC2 using Terraform requires a few prerequisites.
• AWS Account: You need an AWS account to create and manage resources like EC2 instances.
• Terraform Installed: Install Terraform on your local machine to run Terraform commands. Ensure its configured properly and accessible from the command line.
• AWS CLI : Having the AWS CLI installed and configured can be helpful for managing AWS resources and troubleshooting.

The Terraform codes for this project can be found in the link below.
https://github.com/Samuel7050/jenkins-server-terraform.git

Terraform Codes Review
The Terraform codes consist of the modules folder, script folder, main.tf, local.tf and terraform.tf files.

Modules Folder

ec2 folder – This folder contains the main.tf, output.tf and variable.tf files that defines the ec2 resource and its dependencies. This module defines the following -
• security group

• keypairs

• Launch Instance

Iam folder- This module defines the IAM policy, IAM Role, IAM policy attachment and IAM Instance profile for the EC2 server.

ssh_connection folder- This module defines a null resource that will enable ssh connection into our EC2 instance, copy script.sh file from the local machine into the EC2 instance, set permissions and execute the shell script in the script folder.

vpc folder- This module defines a default VPC. A default vpc and default subnet will be created if they are not currently existing.

Script folder- This folder contains the script.sh file. This script contains the required tools that will be installed on the Jenkins server at launch.

Local.tf- This file allows us to pass values to all variables as required in the terraform code.

main.tf- This is the main terraform file that will be used for deploying our infrastructures by referencing the modules folder.

terraform.tf- This is terraform file defines the terraform provider, and configures the aws provider.

Building Infrastructure with Terraform codes.
Open the terminal in the root of your project folder.

Run terraform init to initialize terraform.

Run terraform plan to preview the infrastructure to deploy.

Run terraform apply to deploy the infrastructure on AWS.

*Confirm Infrastructure on AWS *

To access the Jenkins server, copy and paste the output of the website_url on your web browser.

The admin password can be retrieved from the terraform outputs on the terminal.

or from the location on the Jenkins server
/var/lib/jenkins/secrets/initialAdminPassword

Install suggested plugins and configure the admin user credentials.

At this point, we have successfully deployed our Jenkins server on an EC2 instance, and we are ready to create our first CICD pipeline.
In the next project, we will use this Jenkins server to create a CICD pipeline that pulls code from git , build, and deploy applications on AWS.

Watch this space!!!

Clean Up: To avoid bills from AWS, you can destroy the infrastructure by running terraform destroy command.
I will be using this server for subsequent project; hence, I will just stop the instance from the AWS console and start it anytime I need to use it.
Thanks for reading!!!

Deploying an E-commerce Website on Amazon EKS Using Terraform

Gbenga Ojo-Samuel — Sun, 28 Jan 2024 22:37:51 +0000

Overview:
Amazon Elastic Kubernetes Service (Amazon EKS) is a managed Kubernetes service that makes it easy for you to run Kubernetes on AWS and on-premises. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Amazon EKS is certified Kubernetes-conformant, so existing applications that run on upstream Kubernetes are compatible with Amazon EKS.
Amazon EKS automatically manages the availability and scalability of the Kubernetes control plane nodes responsible for scheduling containers, managing application availability, storing cluster data, and other key tasks.
Amazon EKS lets you run your Kubernetes applications on both Amazon Elastic Compute Cloud (Amazon EC2) and AWS Fargate. With Amazon EKS, you can take advantage of all the performance, scale, reliability, and availability of AWS infrastructure, as well as integrations with AWS networking and security services, such as application load balancers (ALBs) for load distribution, AWS Identity and Access Management (IAM) integration with role-based access control (RBAC), and AWS Virtual Private Cloud (VPC) support for pod networking.
Deploying an e-commerce web application on Amazon EKS using Terraform involves several steps. Below is a step-by-step guide to help you set up an EKS cluster and deploy a simple e-commerce web app.

Step 1: Set Up Terraform Configuration
Check the link below for the full terraform configurations.

Terraform Codes Repo
https://github.com/Samuel7050/EKS-Terraform--Project.git

Container Repo
https://hub.docker.com/r/ojosamuel/e-commerce

We will be using Terraform modules for this deployment. Modules are a key feature of Terraform that promotes modularity, reusability, and maintainability in infrastructure as code.

The following infrastructure will be deployed using the terraform Modules.

VPC– The Terraform codes in the vpc module will deploy a default VPC and default subnets in the selected region if it doesn’t exit already. If it does exist, it maintains the default VPC and Subnets.

Security Group– The Terraform codes in the security group module will deploy a security group for the EC2 instance opening port 22 for ssh. This will enable access into the EC2 instance for further interaction with the EKS cluster.

IAM Roles– The IAM Roles Module will deploy the required policy permissions for the EC2 instance, Amazon EKS Cluster and the EKS worker node group. These Roles and attached policies are very vital to ensure that our EKS cluster are working perfectly.

EKS and Worker Node Group– The Terraform codes in the EKS Module will deploy the EKS cluster and the Worker Node Group.

EC2 Instance– The Terraform codes in the EC2 module will deploy an EC2 instance that will serve as the K8-Client Server for interacting with our Amazon EKS Cluster and the Worker Nodes.

User Data– We will be using the script in ec2.sh to preinstall Docker, AWS CLI , Git and Kubectl on the Ec2 instance.

Step 2: Initialize and Apply Terraform Configuration
Run terraform Init – to initialize terraform.

Run terraform plan – To preview all resources to be deployed.

Run terraform apply – To deploy all resources.

Terraform will create the necessary AWS resources as indicated in the main.tf file.

*Checking our Resources on AWS *

IAM Roles

EKS Cluster and Worker Node Group

EC2 Instance

Step 3: Confirm Preinstalled tools and configure **kubectl to Use the EKS Cluster**
Connect to the EC2 instance: I will be connecting to EC2 instance with my VScode IDE. ( compare the internal IP of the EC2 as shared above)

Confirm preinstalled tools are up and running.

Authenticate with your AWS credentials.
Use your Access key ID and your secret access key with programmatic access to authenticate. Note- Do not share your secrets with anyone.

configure kubectl to Use the EKS Cluster

Run aws eks --region us-east-2 update-kubeconfig --name eks-cluster1 to configure kubectl

Run kubectl get nodes command to confirm we have access to our EKS cluster worker nodes.

No resource currently running in the cluster.

Step 4: Deploy a Sample E-Commerce Web App

Create both deployment.yaml file and service.yaml file

Create Deployment: Run kubectl apply -f deployment.yaml command. This will create a deployment , a ReplicaSet and the specified number of Pods.

Create Service: Run kubectl apply -f service.yaml command. This will create a Load balancer type of service. A load balancer will be deployed immediately on AWS.

Load Balancer.

To access our e-commerce website, copy the load balancer dns endpoint url and paste it on your web browser

Congratulations!!! We have successfully deployed our e-commerce website on Amazon EKS.

Thanks for Reading

Clean Up
Run terraform destroy to remove all infrastructures deployed to avoid unexpected bills