DEV Community: Maurice Makafui A.

Creating a Weather Dashboard with AWS S3 buckets and OpenWeather API

Maurice Makafui A. — Tue, 07 Jan 2025 12:55:21 +0000

Achieving the Weather Dashboard Project

This is a comprehensive guide on how I successfully completed the Weather Dashboard project, utilizing Python, AWS S3, and the OpenWeather API. By following these steps, I learnt how to interact with APIs, set up a Python environment, configure AWS CLI, and handle files with AWS S3.

Create an OpenWeather Account and Obtain API Key
Step:
• I visited the OpenWeather website and created an account.
• After logging in, I navigated to the API section and generated an API key. This key was used to fetch weather data for different regions.
Lesson Learnt:
• API Integration: I learnt how to create and use an API key to interact with external APIs (like OpenWeather). The API key is a critical part of securely accessing the data provided by the service.
Set Up My Work Environment in VSCode
Step:
• I created a new directory for the project and opened it in VSCode.
• Inside the project directory, I created a .env file. This file will store sensitive information like the OpenWeather API key and the AWS S3 bucket name.
o Example:

WEATHER_API_KEY=your_openweather_api_key
AWS_BUCKET_NAME=unique_bucket_name

Lesson Learnt:
• Environment Setup: I learnt how to use .env files to securely store sensitive data such as API keys. This is a good practice to avoid hardcoding sensitive information in my source code.

Set Up AWS in My Terminal
Step:
• I logged into my AWS console and navigated to IAM (Identity and Access Management).
• I created a new IAM user and attached the "AdministratorAccess" and "AmazonS3FullAccess" policies.
• I generated an access key and secret key for the new IAM user.
• In VSCode terminal, I ran aws configure and entered the Access Key ID and Secret Access Key when prompted.
Lesson Learnt:
• AWS CLI Configuration: I learnt how to configure AWS CLI using IAM credentials. This allows me to interact with AWS services (like S3) directly from my terminal using Python.
Set Up My Python Environment
Step:
• I downloaded and installed Python on my machine (Windows or Linux, based on my system).
• I set up environment variables as needed.
Lesson Learnt:
• Python Setup: I gained experience in setting up a Python environment on my machine, including installing Python and configuring the system for running Python scripts.

Folder Structure Setup Step: • I organized my project folder structure as follows: • weather-dashboard/ • `

├─ src/
    │   ├── __init__.py
    │   └── weather_dashboard.py
    ├── .env
    ├── .gitignore
    └── requirements.txt``

I added .gitignore to prevent sensitive files from being tracked by Git:

•

.env
• .git

• In the requirements.txt, I listed the necessary Python packages needed for the smooth run:
•

boto3==1.26.137
• python-dotenv==1.0.0
• requests==2.28.2

Lesson Learnt:
• Project Organization: I learnt how to structure a Python project to ensure clarity and maintainability. This also includes using. gitignore to keep sensitive files out of version control.

Install Required Python Packages
Step:
• I ran the following command to install the packages in requirements.txt:
• python -m pip install -r requirements.txt
Lesson Learnt:
• Dependency Management: I learnt how to manage project dependencies with requirements.txt. This ensures all the required libraries are installed in one go.
Write the Script to Fetch and Store Weather Info
Step:
•

    I imported the required libraries:
• import requests
• import boto3
• from dotenv import load_dotenv
• import json
• import os

• I fetched weather data from the OpenWeather API and parsed it into JSON format.
• I created an S3 bucket and uploaded the weather data as a JSON file.
Lesson Learnt:
• API Data Handling: I learnt how to fetch and process data from an API and parse it into a format suitable for storage (JSON). This skill is essential for working with many types of external data sources.
• AWS S3 Integration: I learnt how to interact with AWS S3 to upload files from Python using boto3. This is crucial for cloud storage automation.

Run the Python Script
Step:
• After writing the script, I navigated to the src folder and ran the following command:
• python weather_dashboard.py
Lesson Learnt:
• Testing and Debugging: I learnt the importance of testing my code regularly to ensure it works as expected. Running the script helped verify that all components were correctly integrated.
Final Check and Housekeeping
Step:
• After running the script, the terminal should show:
o The S3 bucket is created successfully.
o Weather data is displayed in the terminal and uploaded to the S3 bucket.
• I went to my AWS console and checked the S3 bucket to confirm the presence of the uploaded files.
• Once verified, I emptied the bucket and deleted it to prevent unnecessary charges.
Lesson Learnt:
• Resource Cleanup: I learnt the importance of cleaning up cloud resources (like S3 buckets) after use to avoid unexpected costs. This is an important best practice when working with cloud services.

Summary of Lessons Learnt

Working with APIs: I learnt how to fetch data from an external API (OpenWeather) and parse it into a format usable by my application.
Environment Setup: I gained experience in setting up a Python development environment, including managing sensitive data with .env files.
AWS CLI Configuration: I learnt how to configure AWS CLI with IAM credentials, allowing me to interact with AWS services such as S3 directly from the terminal.
Folder Structure and Project Organization: I understood the importance of structuring my project correctly and using. gitignore to keep sensitive data secure.
Python Package Management: I installed and managed dependencies using pip and requirements.txt to ensure my project runs smoothly.
AWS S3 Integration: I learnt how to use the boto3 library to interact with AWS S3, including creating buckets and uploading files.
Error Handling and Debugging: I gained skills in debugging common issues like missing credentials or AWS permission errors.
Resource Cleanup: I learnt to clean up AWS resources after use to avoid unnecessary billing and to keep the project efficient.

By completing this project, I gained hands-on experience working with APIs, AWS S3, and Python, and learnt key cloud computing practices. I’m now equipped with the knowledge to integrate and automate cloud-based services in my future projects!

Automating User Managment system with Bash: in a linux environment.

Maurice Makafui A. — Wed, 03 Jul 2024 07:41:28 +0000

Onboarding several new workers can make managing users on a Linux system a repetitious chore. This procedure may be automated to save time and lower the possibility of human mistake. This tutorial will walk you through creating a bash script that creates users, assigns them to groups, configures their home directories, and generates passwords based on information retrieved from a text file. We'll also make sure that passwords are safely saved and that all activities are recorded.

Basic requirements

a Linux system having root (administrative) rights.
basic familiarity with bash scripting.

3.A text editor for creating and editing files (such as vim or nano)

How to Use the Script
Follow these steps to use the create_users.sh script:

Clone the Repository: Start by cloning the GitHub repository to your local machine or server.

git clone https://github.com/Maurice-Makafui/STAGE_1_HNG_11.git
cd STAGE_1_HNG_11

Prepare the Input File: Create a text file with the desired usernames and groups. Each line should be formatted as username;group1,group2,group3. Here’s an example:

plaintext

Maurice1;staging,development,deployment
Gwenny;prayergroup
Felix;fitness,gymgroup

Run the Script: Execute the script with the input file as an argument.

sudo bash ./create_users.sh users.txt

Verify the Results:

Check the Passwords:

cat /var/secure/user_passwords.csv

Check the Log File:

cat /var/log/user_management.log

List Users and Groups:

cat /etc/passwd
cat /etc/group

Verify Home Directories:

cd /home && ls

Check Group Membership:

getent group dev

Here is what the Bash Script looks like

#!/bin/bash

# Log file location
LOGFILE="/var/log/user_management.log"
PASSWORD_FILE="/var/secure/user_passwords.csv"

# Ensure the script is run as root
if [[ "$EUID" -ne 0 ]]; then
  echo "Please run as root"
  exit 1
fi

# Check if the input file is provided
if [ -z "$1" ]; then
  echo "Error: No file was provided"
  echo "Usage: $0 <name-of-text-file>"
  exit 1
fi

# Create log and password files
mkdir -p /var/secure
touch "$LOGFILE" "$PASSWORD_FILE"
chmod 600 "$PASSWORD_FILE"

# Function to generate a random password
generate_random_password() {
    local length=${1:-12} # Default length is 12 if no argument is provided
    LC_ALL=C tr -dc 'A-Za-z0-9!?%+=' < /dev/urandom | head -c $length
}

# Function to create a user
create_user() {
  local username=$1
  local groups=$2

  if getent passwd "$username" > /dev/null; then
    echo "User $username already exists" | tee -a "$LOGFILE"
  else
    useradd -m -g "$username" -s /bin/bash "$username"
    echo "Created user $username" | tee -a "$LOGFILE"
  fi

  # Create the user's personal group
  if ! getent group "$username" > /dev/null; then
    groupadd "$username"
    echo "Created group $username" | tee -a "$LOGFILE"
  fi

  # Add user to specified groups
  IFS=',' read -r -a groups_array <<< "$groups"
  for group in "${groups_array[@]}"; do
    if ! getent group "$group" >/dev/null; then
      groupadd "$group"
      echo "Created group $group" | tee -a "$LOGFILE"
    fi
    usermod -aG "$group" "$username"
    echo "Added user $username to group $group" | tee -a "$LOGFILE"
  done

  # Set up home directory permissions
  chmod 700 /home/"$username"
  chown "$username:$username" /home/"$username"
  echo "Set up home directory for user $username" | tee -a "$LOGFILE"

  # Generate a random password
  password=$(generate_random_password)
  echo "$username:$password" | chpasswd
  echo "$username,$password" >> "$PASSWORD_FILE"
  echo "Set password for user $username" | tee -a "$LOGFILE"
}

# Read the input file and create users
while IFS=';' read -r username groups; do
  # Skip empty lines
  if [[ -z "$username" ]]; then
    continue
  fi
  create_user "$username" "$groups"
done < "$1"

echo "User creation process completed." | tee -a "$LOGFILE"

Heres the breakdown

Shebang and Variable Definitions

#!/bin/bash

# Log file location
LOGFILE="/var/log/user_management.log"
PASSWORD_FILE="/var/secure/user_passwords.csv"

!/bin/bash: Specifies that the script should be run with the Bash shell.

LOGFILE: Path to the log file where script actions will be recorded.

PASSWORD_FILE: Path to the file where usernames and passwords will be saved.

Check for Root Privileges

# Ensure the script is run as root
if [[ "$EUID" -ne 0 ]]; then
  echo "Please run as root"
  exit 1
fi

This section checks if the script is run as the root user. EUID is the effective user ID, and 0 corresponds to the root user. If not root, the script prints an error message and exits.

Check for Input File Argument

# Check if the input file is provided
if [ -z "$1" ]; then
  echo "Error: No file was provided"
  echo "Usage: $0 <name-of-text-file>"
  exit 1
fi

Checks if a filename argument is provided when running the script. If not, it prints an error message and shows the correct usage.

Create Log and Password Files

# Create log and password files
mkdir -p /var/secure
touch "$LOGFILE" "$PASSWORD_FILE"
chmod 600 "$PASSWORD_FILE"

Creates the directory /var/secure if it does not exist.
Creates the log file and password file.
Sets the permissions of PASSWORD_FILE to 600 (read and write only for the owner).

Generate Random Password Function

# Function to generate a random password
generate_random_password() {
    local length=${1:-12} # Default length is 12 if no argument is provided
    LC_ALL=C tr -dc 'A-Za-z0-9!?%+=' < /dev/urandom | head -c $length
}

generate_random_password(): A function that generates a random password of a specified length (default is 12 characters).
tr -dc 'A-Za-z0-9!?%+=' < /dev/urandom: Filters out characters from /dev/urandom.
head -c $length: Limits the output to the desired length.

Create User Function

# Function to create a user
create_user() {
  local username=$1
  local groups=$2

  if getent passwd "$username" > /dev/null; then
    echo "User $username already exists" | tee -a "$LOGFILE"
  else
    useradd -m -g "$username" -s /bin/bash "$username"
    echo "Created user $username" | tee -a "$LOGFILE"
  fi

  # Create the user's personal group
  if ! getent group "$username" > /dev/null; then
    groupadd "$username"
    echo "Created group $username" | tee -a "$LOGFILE"
  fi

  # Add user to specified groups
  IFS=',' read -r -a groups_array <<< "$groups"
  for group in "${groups_array[@]}"; do
    if ! getent group "$group" >/dev/null; then
      groupadd "$group"
      echo "Created group $group" | tee -a "$LOGFILE"
    fi
    usermod -aG "$group" "$username"
    echo "Added user $username to group $group" | tee -a "$LOGFILE"
  done

  # Set up home directory permissions
  chmod 700 /home/"$username"
  chown "$username:$username" /home/"$username"
  echo "Set up home directory for user $username" | tee -a "$LOGFILE"

  # Generate a random password
  password=$(generate_random_password)
  echo "$username:$password" | chpasswd
  echo "$username,$password" >> "$PASSWORD_FILE"
  echo "Set password for user $username" | tee -a "$LOGFILE"
}

create_user(): A function to add a new user with specific groups and set up the home directory.
Check if User Exists: Checks if the user already exists. If not, creates the user.
Create Personal Group: Checks if a group named after the user exists. If not, creates it.
Add User to Groups: Parses the groups from the input and adds the user to these groups.
Home Directory Setup: Sets the correct permissions and ownership for the user’s home directory.
Generate and Set Password: Generates a password, updates the user’s password, and logs it.

Read Input File and Create Users

# Read the input file and create users
while IFS=';' read -r username groups; do
  # Skip empty lines
  if [[ -z "$username" ]]; then
    continue
  fi
  create_user "$username" "$groups"
done < "$1"

Reads the input file line by line.
Each line should contain a username and groups, separated by a semicolon.
Calls create_user for each valid line.

Final Message

echo "User creation process completed." | tee -a "$LOGFILE"

Prints a message indicating that the user creation process is complete and logs it.

How Everything Works
1.Configuration: The directories for the password and log files are specified by the script.

2.User File Check: It determines whether the user file is attached and is there.

3.Setup: Makes the required files and folders with the right permissions.
4.Compiling Every User: reads every line in the file, handles the groups and username, and then carries out the following operations

5.Finalization: Documents the accomplishment of the user creation procedure.

Conclusion
On a Linux system, you may automate the process of generating users and grouping them by using this method. During the user creation process, this script helps to maintain consistency, saves time, and lowers the possibility of mistakes. Please feel free to make any changes to the script to suit your needs.

Acknowledgments

I would want to express my appreciation to HNG Hire for giving me the chance and means to create this solution.
Explore resources and programs such as the HNG Internship and HNG Premium for more advanced subjects and automation tips.