DEV Community: Rene Francisco Cruz González

Building a Weather Data Analytics Pipeline with AWS and OpenWeatherMap API

Rene Francisco Cruz González — Fri, 17 Jan 2025 04:25:02 +0000

Hey coders! In this blog post, I will walk you through the process of building a weather data analytics pipeline using the OpenWeatherMap API and various AWS services. This project involves fetching weather data, storing it in an S3 bucket, cataloging it using AWS Glue, and querying it with Amazon Athena.

Project Overview

The goal of this project is to create a scalable and efficient data pipeline that can fetch weather data for multiple cities, store it in AWS S3, catalog the data using AWS Glue, and perform queries using Amazon Athena.

Initial architecture design

Architecture

Project Structure

Prerequisites

Before you begin, ensure you have the following:

Docker: Installed on your machine.
AWS Account: With permissions to create S3 buckets, Glue databases, and Glue crawlers.
OpenWeatherMap API Key: Obtain an API key from OpenWeatherMap.

Setup Instructions

Step 1: Clone the Repository

First, clone the repository and navigate to the project directory:

git clone https://github.com/Rene-Mayhrem/weather-insights.git
cd weather-data-analytics

Step 2: Create a .env File

Create a .env file in the root directory with your AWS credentials and OpenWeatherMap API key:

AWS_ACCESS_KEY_ID=<your-access-key-id>
AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
AWS_REGION=us-east-1
S3_BUCKET_NAME=<your-s3-bucket-name>
OPENWEATHER_API_KEY=<your-openweather-api-key>

Step 3: Create a cities.json File

Create a cities.json file in the root directory with a list of cities to study:

{
  "cities": [
    "London",
    "New York",
    "Tokyo",
    "Paris",
    "Berlin"
  ]
}

Step 4: Using Docker Compose

Build and run the services using Docker Compose:

docker compose run terraform init
docker compose run python

Usage

After running the Docker containers, follow these steps:

Verify Infrastructure Setup

Ensure that Terraform has successfully created the necessary AWS resources (S3 bucket, Glue database, and Glue crawler). You can verify this in the AWS Management Console.

Verify Data Upload

Check that the Python script has fetched weather data for the specified cities and uploaded the data to the S3 bucket. Verify the JSON files in the S3 bucket via the AWS Management Console.

Run the Glue Crawler

The Glue crawler should automatically run if set up correctly. This will catalog the data in the S3 bucket. Verify the crawler's run and data cataloging in the Glue console.

Query Data with Athena

Use Amazon Athena to query the data cataloged by Glue. Access Athena through the AWS Management Console and run SQL queries on the data.

Key Interactions

Docker: Provides two consistent environments for running scripts and Iac (Python and terraform).
Terraform: Provisions and manage AWS resources (S3, AWS Glue, AWS Athena, etc).
Python: Fetch weather data and create JSON files to upload them to the S3 bucket created with terraform.
Glue: Crawls S3 data to catalog data and create schemas.
Athena: Queries data for insights.

Conclusion

By following these steps, you can set up a robust weather data analytics pipeline using AWS services and the OpenWeatherMap API. This pipeline can be extended to include more cities or additional data sources as needed.

Cryptocurrency Notifications System Alert

Rene Francisco Cruz González — Fri, 10 Jan 2025 00:37:15 +0000

Hey Coders, welcome to my blog. Today, I’m excited to share a project I’ve been working on: the Cryptocurrency Notifications Alert System. This project leverages AWS services, Terraform, and Docker to fetch cryptocurrency data from an external API and send notifications via Amazon SNS.

Check my code at: https://github.com/Rene-Mayhrem/Crypto-Notifications
Follow me in LinkedIn if you want to connect: https://www.linkedin.com/in/renecruz-1202r/

Project Overview

The Cryptocurrency Notifications Alert System is designed to keep you updated on the latest cryptocurrency prices and market data. It fetches data from the CoinGecko API and sends notifications about the cryptocurrencies' status, prices, and other details via Amazon SNS.

Architecture Diagram

Here’s an overview of the system architecture:

Docker Containers: Used to run Terraform commands and package the Lambda function.
Terraform: Provisions AWS resources such as IAM roles, SNS topics, and Lambda functions.
AWS Services: Includes IAM, SNS, and Lambda to handle notifications and function execution.

Instructions

Clone the repository:

git clone https://github.com/Rene-Mayhrem/Crypto-Notifications.git
cd Crypto-Notifications

Create a zip ofthe Lambda function.
- lambda_function.py: Contains the lambda implementatino code.
- coins.txt: Contains a list of all rthe currencies to be analized

cd src
zip lambda_function.zip lambda_function.py coins.txt

Create Environment variables: Create a .env file in the root directory with the following content:

API_KEY=your_api_key
AWS_ACCESS_KEY_ID=your_aws_access_key_id
AWS_SECRET_ACCESS_KEY=your_aws_secret_access_key
AWS_ACCOUNT_ID=your_aws_account_id
AWS_REGION=your_aws_region
AWS_TOPIC=crypto_notifications

Create terraform.tfvars for variables: Create a terraform.tfvars file in the root directory with the following content:

region = "your_aws_region"
account_id = "your_aws_account_id"
topic = "crypto_notifications"

Add your email to the resource at main.tf line 49, where the email_subscription is created, in the endpoint part:

resource "aws_sns_topic_subscription" "email_subscription" {
  topic_arn = aws_sns_topic.crypto_notifications.arn
  protocol  = "email"
  endpoint  = "your-email"
}

Project Structure:

The prioject direcotry is as follows:

.
├── .env
├── docker-compose.yml
├── policies
│   └── gd_notifications.json
├── src
│   ├── coins.txt
│   ├── lambda_function.py
│   └── lambda_function.zip
├── main.tf
├── terraform.tfvars
└── README.md

Terraform configuration

The Terraform configuration is used to provision the necessary AWS resources. Here’s a brief overview of the key resources:

IAM Role: Allows the Lambda function to publish messages to the SNS topic.
SNS Topic: Receives notifications about cryptocurrency updates.
Lambda Function: Executes the code to fetch cryptocurrency data and send notifications.

Lambda Function

The Lambda function fetches cryptocurrency data from the CoinGecko API and sends notifications via Amazon SNS. Here’s the code:

import os
import json
import urllib.request
import boto3

def format_crypto_data(crypto):
    # Extract cryptocurrency details from the API response
    name = crypto.get("name", "Unknown")
    symbol = crypto.get("symbol", "Unknown")
    current_price = crypto.get("market_data", {}).get("current_price", {}).get("usd", "Unknown")
    market_cap = crypto.get("market_data", {}).get("market_cap", {}).get("usd", "Unknown")
    total_volume = crypto.get("market_data", {}).get("total_volume", {}).get("usd", "Unknown")

    # Format the extracted data into a readable message
    return (
        f"Cryptocurrency Update for {name} ({symbol}):\n"
        f"Current Price: ${current_price}\n"
        f"Market Cap: ${market_cap}\n"
        f"Total Volume: ${total_volume}\n"
    )

def lambda_handler(event, context):
    # Get the SNS topic ARN from environment variables
    sns_topic_arn = os.getenv("SNS_TOPIC_ARN")
    # Initialize the SNS client
    sns_client = boto3.client("sns")

    # Path to the file containing cryptocurrency IDs
    coins_file_path = "./coins.txt"
    # Read the cryptocurrency IDs from the file
    with open(coins_file_path, "r") as file:
        crypto_ids = [line.strip() for line in file.readlines()]

    # List to store formatted messages
    messages = []

    # Fetch data for each cryptocurrency ID
    for crypto_id in crypto_ids:
        api_url = f"https://api.coingecko.com/api/v3/coins/{crypto_id}"
        try:
            # Make a request to the CoinGecko API
            with urllib.request.urlopen(api_url) as response:
                # Parse the JSON response
                data = json.loads(response.read().decode())
                # Format the data into a readable message
                message = format_crypto_data(data)
                # Add the message to the list
                messages.append(message)
        except Exception as e:
            # Print an error message if the API request fails
            print(f"Error fetching data for {crypto_id}: {e}")

    # Publish each message to the SNS topic
    for message in messages:
        try:
            sns_client.publish(
                TopicArn=sns_topic_arn,
                Message=message,
                Subject="Cryptocurrency Update"
            )
        except Exception as e:
            # Print an error message if publishing to SNS fails
            print(f"Error publishing message to SNS: {e}")

    # Return a success response
    return {
        "statusCode": 200,
        "body": json.dumps("Data processed and sent to SNS")
    }

Running the prioject

Initialize Terraform:

docker-compose run terraform_container init

Apply the Terraform configuration:

docker-compose run terraform_container apply -auto-approve

Deploy and test your lambda
- Go to AWS Lambda
- Open your created Lambda
- Create a test

Check your email

Destroy the Terraform-managed infrastructure (if needed):

docker-compose run terraform_container destroy -auto-approve

Conclusion

The Cryptocurrency Notifications Alert System is a powerful tool to stay updated on the latest cryptocurrency prices and market data. By leveraging AWS services, Terraform, and Docker, this project provides a scalable and efficient solution for real-time notifications. Future improvements could include adding more cryptocurrencies, enhancing the notification format, and integrating with other messaging platforms.

I hope you found this blog post helpful. Feel free to reach out with any questions or feedback!

Game Notification Altert System with AWS Lambda and AWS SNS

Rene Francisco Cruz González — Wed, 08 Jan 2025 19:16:17 +0000

Hey Coders! Today, I am excited to share a project that combines the power of AWS Lambda, Amazon SNS, and an external API to create a robust NBA Game Notifications Alert System. This system fetches real-time NBA game data and sends notifications about the games' status, scores, and other details directly to your email or any other subscribed endpoint.

In this blog, I will walk you through the entire process of setting up this notification system, from cloning the repository to deploying the Lambda function and configuring the necessary AWS services. Whether you are a sports enthusiast looking to stay updated with the latest NBA games or a developer interested in learning more about AWS services, this project has something for you.

Let's dive in and explore how to build this exciting notification system step-by-step!

NBA Game Notifications Alert System
- Table of Contents
- Overview
- Architecture Diagram
- Setup
- Lambda Function

Prerequisites

AWS Account
Free Subscription in NBA API

Overview

The NBA Game Notifications Alert System fetches NBA game data from an external API and sends notifications about the games' status, scores, and other details via Amazon SNS.

Architecture Diagram

Setup

Clone the repository:

git clone <repository-url>
cd Game-Notification-with-AWS-Lambda-Amazon-SNS-and-AmazonBridge

Create SNS Topic and Subscriber:
- In AWS, go to SNS and create a new topic named gd_topic.
- Create a new subscription, choose email as the protocol, and select the ARN of the gd_topic.

Create IAM Policy and Role:

In AWS, go to IAM and create a new policy with the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": "sns:Publish",
          "Resource": "arn:aws:sns:REGION:ACCOUNT_ID:TOPIC"
      }
  ]
}

- Create a new role, select AWS Services, and assign the following policies:
    - The SNS publish policy created above.
    - Lambda Basic Execution Role.
- Name the role and create it.

Create Lambda Function:
- In AWS Lambda, create a new function from scratch.
- Enter a function name and assign the previously created role.
- Copy the code from src/lambda_function.py into the function.
- Add the following environment variables:
```
NBA_API_KEY=Your_API_Key
SNS_TOPIC_ARN=Your_SNS_Topic_ARN
```

- Create the function.

Deploy and Test Your Lambda Function:
- Open the Lambda function and create a new test event.
- Run the function and check the logs in CloudWatch for any errors.

Lambda Function

The main logic of the project is implemented in the lambda_function.py file located in the src directory. Here is a brief overview of the key components:

Imports:

import os
import json
import urllib.request
import boto3
from datetime import datetime, timedelta, timezone

Function to Format Game Data:

def format_game_data(game):
    # Get environment variables
  api_key = os.getenv("NBA_API_KEY")
  sns_topic_arn = os.getenv("SNS_TOPIC_ARN")
  sns_client = boto3.client("sns")

  # Adjust for Central Time (UTC-6)
  utc_now = datetime.now(timezone.utc)
  central_time = utc_now - timedelta(hours=6)  # Central Time is UTC-6
  today_date = central_time.strftime("%Y-%m-%d")

  print(f"Fetching games for date: {today_date}")

  # Fetch data from the API
  api_url = f"https://api.sportsdata.io/v3/nba/scores/json/GamesByDate/{today_date}?key={api_key}"
  print(today_date)

  try:
      with urllib.request.urlopen(api_url) as response:
          data = json.loads(response.read().decode())
          print(json.dumps(data, indent=4))  # Debugging: log the raw data
  except Exception as e:
      print(f"Error fetching data from API: {e}")
      return {"statusCode": 500, "body": "Error fetching data"}

  # Include all games (final, in-progress, and scheduled)
  messages = [format_game_data(game) for game in data]
  final_message = "\n---\n".join(messages) if messages else "No games available for today."

  # Publish to SNS
  try:
      sns_client.publish(
          TopicArn=sns_topic_arn,
          Message=final_message,
          Subject="NBA Game Updates"
      )
      print("Message published to SNS successfully.")
  except Exception as e:
      print(f"Error publishing to SNS: {e}")
      return {"statusCode": 500, "body": "Error publishing to SNS"}

  return {"statusCode": 200, "body": "Data processed and sent to SNS"}

Lambda Handler Function:

  def lambda_handler(event, context):
    status = game.get("Status", "Unknown")
    away_team = game.get("AwayTeam", "Unknown")
    home_team = game.get("HomeTeam", "Unknown")
    final_score = f"{game.get('AwayTeamScore', 'N/A')}-{game.get('HomeTeamScore', 'N/A')}"
    start_time = game.get("DateTime", "Unknown")
    channel = game.get("Channel", "Unknown")

    # Format quarters
    quarters = game.get("Quarters", [])
    quarter_scores = ', '.join([f"Q{q['Number']}: {q.get('AwayScore', 'N/A')}-{q.get('HomeScore', 'N/A')}" for q in quarters])

    if status == "Final":
        return (
            f"Game Status: {status}\n"
            f"{away_team} vs {home_team}\n"
            f"Final Score: {final_score}\n"
            f"Start Time: {start_time}\n"
            f"Channel: {channel}\n"
            f"Quarter Scores: {quarter_scores}\n"
        )
    elif status == "InProgress":
        last_play = game.get("LastPlay", "N/A")
        return (
            f"Game Status: {status}\n"
            f"{away_team} vs {home_team}\n"
            f"Current Score: {final_score}\n"
            f"Last Play: {last_play}\n"
            f"Channel: {channel}\n"
        )
    elif status == "Scheduled":
        return (
            f"Game Status: {status}\n"
            f"{away_team} vs {home_team}\n"
            f"Start Time: {start_time}\n"
            f"Channel: {channel}\n"
        )
    else:
        return (
            f"Game Status: {status}\n"
            f"{away_team} vs {home_team}\n"
            f"Details are unavailable at the moment.\n"
        )

The lambda_handler function performs the following steps:

Retrieves environment variables.
Adjusts the current time to Central Time (UTC-6).
Fetches NBA game data from the external API.
Formats the game data.
Publishes the formatted data to the specified SNS topic.

Building a Weather Dashboard with Docker and AWS S3

Rene Francisco Cruz González — Mon, 06 Jan 2025 17:07:56 +0000

Hey Coders! In this blog post, we'll walk through the contents of a repository that demonstrates how to build a weather dashboard application using Docker and AWS S3. This application fetches weather data from the OpenWeather API and saves it to an AWS S3 bucket. Let's dive into the details!

Here's my github repository, feel free to check it: weather-app

If you want to connect follow me on Github: Rene-Mayhrem

Repository Structure

Here's an overview of the repository structure:

weather-dashboard/
├── src/
│   ├── __init__.py
│   └── weather_dashboard.py
├── .env
├── .gitignore
├── Dockerfile
└── README.md

Key Files and Their Purpose

1. init.py

This file serves as the entry point for the application. It imports and calls the main function from weather_dashboard.py, ensuring the application runs when the Docker container starts.

from weather_dashboard import main

if __name__ == "__main__":
    main()

2. weather_dashboard.py

This is the main application file. It contains the WeatherDashboard class, which handles fetching weather data from the OpenWeather API and saving it to an AWS S3 bucket. The class also includes methods for creating the S3 bucket if it doesn't exist and saving weather data to the bucket.

3. .env

This file contains environment variables required by the application, such as the OpenWeather API key and AWS credentials. It ensures sensitive information is not hard-coded into the application.

OPENWEATHER_API_KEY=your_openweather_api_key
AWS_BUCKET_NAME=weather-dashboard-${RANDOM}
AWS_ACCESS_KEY_ID=your_aws_access_key_id
AWS_SECRET_ACCESS_KEY=your_aws_secret_access_key
AWS_DEFAULT_REGION=your_aws_default_region

4. .gitignore

This file specifies which files and directories should be ignored by Git. It includes entries for the

.env

file, Python cache directories, and other unnecessary files.

.env
__pycache__/
*.zip
test/
data/

5. Dockerfile

The Dockerfile defines the steps to build a Docker image for the application. It installs the necessary dependencies, copies the application code, and sets the command to run the application.

FROM python:3.9-slim

WORKDIR /app 

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

# Install AWS CLI
RUN apt-get update && apt-get install -y awscli

COPY .env .
COPY src/ ./src  

CMD ["python", "src/__init__.py"]

6. README.md

The README file provides detailed instructions on how to set up and run the application. It includes sections on prerequisites, setup steps, usage, AWS configuration, and resources for further reading.

Setting Up the Application

To set up the application, follow these steps:

Clone the repository:

git clone https://github.com/Rene-Mayhrem/aws-weather-app.git
cd weather-dashboard

Create a .env file with your OpenWeather API key and AWS credentials.
Build the Docker image:
```
docker build -t weather-dashboard .
```

Run the Docker container:

docker run --name aws-app weather-dashboard

Conclusion

This repository provides a comprehensive example of how to build a weather dashboard application using Docker and AWS S3. By following the setup instructions, you can easily fetch weather data and save it to an S3 bucket. The repository structure and detailed documentation make it easy to understand and extend the application for your own needs.

Feel free to comment and discuss.

1768. Merge Strings Alternately

Rene Francisco Cruz González — Mon, 06 Jan 2025 06:12:23 +0000

Hey coders! Hope you're doing well. I'm excited to share my solutions for the LeetCode-75 series, which covers 75 essential problems to help you prepare for coding interviews.

In each post, I'll present my solution along with a detailed explanation of my approach. Feel free to leave any questions or suggestions for improvement in the comments. I'm looking forward to collaborating and discussing with you! Happy Coding!

I've added here the link for the problem: Merge Strings Alternately

Problem description

You are given two strings word1 and word2. Merge the strings by adding letters in alternating order, starting with word1. If a string is longer than the other, append the additional letters onto the end of the merged string.

Return the merged string.

Example 1:

Input: word1 = "abc", word2 = "pqr"
Output: "apbqcr"
Explanation: The merged string will be merged as so:
word1: a b c
word2: p q r
merged: a p b q c r

Example 2:

Input: word1 = "ab", word2 = "pqrs"
Output: "apbqrs"
Explanation: Notice that as word2 is longer, "rs" is appended to the end.
word1: a b
word2: p q r s
merged: a p b q r s

Example 3:

Input: word1 = "abcd", word2 = "pq"
Output: "apbqcd"
Explanation: Notice that as word1 is longer, "cd" is appended to the end.
word1: a b c d
word2: p q
merged: a p b q c d

SOLUTION

Intuition

Given two strings, we need to merge them by alternating characters from each string. The solution is straightforward if both strings have the same length, but they can have different lengths. We will iterate through both strings using pointers, adding characters to the result until both pointers reach the end.

Approach

Create a StringBuilder to store the alternated characters from both strings.
Create two pointers to keep track of the current position in each string.
Iterate both strings until both pointers reach the end of their respective strings.
Add elements to the StringBuilder if the string is not empty and increment the pointer
Return the StringBuilder

Complexity

Time complexity:
The time complexity is O(n) where n is the length of the longer string, as we iterate through the strings.
Space complexity:
The time complexity is 0(1) since we use a StringBuilder and a few variables.

Code

  public String mergeAlternately (String word1, String word2) {
    // ? Create a StringBuilder to build the result string efficiently
    StringBuilder completeWord = new StringBuilder();

    // ? Initialize two pointers to traverse both strings
    int p1 = 0;
    int p2 = 0;

    // ? Iterate through both strings until both pointers reach the end of their resépectives strings
    while (p1 < word1.length() || p2 < word2.length()) {
      // ? Append the current character from words if the pointer is within bounds
      if (p1 < word1.length()) completeWord.append(word1.charAt(p1));
      if (p2 < word2.length()) completeWord.append(word2.charAt(p2));
      p1++;
      p2++;
    }
    // ? Convert the StringBuilder to a string and return it
    return completeWord.toString();
  }

DEV Community: Rene Francisco Cruz González

Building a Weather Data Analytics Pipeline with AWS and OpenWeatherMap API

Project Overview

Initial architecture design

Architecture

Project Structure

Prerequisites

Setup Instructions

Step 1: Clone the Repository

Step 2: Create a .env File

Step 3: Create a cities.json File

Step 4: Using Docker Compose

Usage

Verify Infrastructure Setup

Verify Data Upload

Run the Glue Crawler

Query Data with Athena

Key Interactions

Conclusion

Cryptocurrency Notifications System Alert

Project Overview

Architecture Diagram

Instructions

Project Structure:

Terraform configuration

Lambda Function

Running the prioject

Conclusion

Game Notification Altert System with AWS Lambda and AWS SNS

Table of Contents

Prerequisites

Overview

Architecture Diagram

Setup

Lambda Function

Building a Weather Dashboard with Docker and AWS S3

Repository Structure

Key Files and Their Purpose

1. init.py

2. weather_dashboard.py

3. .env

4. .gitignore

5. Dockerfile

6. README.md

Setting Up the Application

Conclusion

1768. Merge Strings Alternately

Problem description

*Example 1: *

Example 2:

** Example 3:**

SOLUTION

Intuition

Approach

Complexity

Code

Example 1:

Example 3: