DEV Community

Cover image for Store weather data in Amazon S3
Ogooluwa Akinola
Ogooluwa Akinola

Posted on • Edited on

Store weather data in Amazon S3

In this article, we will look at how to store JSON data in an Amazon S3 bucket.

Technologies & Tools

Pre-requisite

Setup Python

if you haven't, please setup Python on you local machine.

[Optional] Github Account & Repo

Skip this if you don't intend to save your project on github. However, you can follow these links to setup a Github Account & Repo.

Setup AWS account and IAM User

Create you AWS Free Tier account if you haven't. Then, follow these steps to create your IAM user. Ensure the IAM user is assigned the AmazonS3FullAccess policy.

IAM-AmazonS3FullAccess

Setup AWS CLI

We are going to be using the Boto3 AWS SDK to interact with AWS. Underneath, the AWS uses a AWS API KEY and secret key to authenticate with AWS. Hence we need to configure AWS credentials on our local machine. Please follow the Boto3 documentation for configuration guide. Ensure the IAM user credentials you're using is configured with the AmazonS3FullAccess policy mentioned above.

OpenWeather API

Openweather is the external api that we will use in this project to provide real time weather data. Please sign up to OpenWeather service to get your OpenWeather API key.

Openweather-api-key

Now let's get started! 😀

P.S the CLI commands examples will be for MacOs/linux users. For windows users you can research the commands online.

Step 1

Create a folder called weather-dashboard-demo

mkdir weather-dashboard-demo
Enter fullscreen mode Exit fullscreen mode

Step 2

Change directory to weather-dashboard-demo and create necessary files and folders

cd weather-dashboard-demo
Enter fullscreen mode Exit fullscreen mode
mkdir src
Enter fullscreen mode Exit fullscreen mode
touch .env requirements.txt
Enter fullscreen mode Exit fullscreen mode

Step 3

Initialize git (optional)

git init
Enter fullscreen mode Exit fullscreen mode

Step 4

Add the package dependencies in re requirements.txt

echo "boto3==1.26" >> requirements.txt
Enter fullscreen mode Exit fullscreen mode
echo "python-dotenv==1.0.0" >> requirements.txt
Enter fullscreen mode Exit fullscreen mode
echo "requests==2.28" >> requirements.txt
Enter fullscreen mode Exit fullscreen mode

Step 5

Install your dependencies

pip3 install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

Step 6

Add env variables to your .env file.

Get your API key from the open weather account you created above.

echo "OPENWEATHER_API_KEY=<replace-with-your-api-key>" >> .env
Enter fullscreen mode Exit fullscreen mode

Add your s3 bucket name. Note your bucket name must be globally unique.

echo "AWS_BUCKET_NAME=weather-dashboard-<append-unique-value>" >> .env
Enter fullscreen mode Exit fullscreen mode

Step 7

Create the necessary files in the src folder

touch src/__init__.py src/weather_dashboard.py
Enter fullscreen mode Exit fullscreen mode

Step 8

Copy the full code into the weather_dashboard.py. Don't feel overwhelmed I will explain the code.

import os
import json
import boto3
import requests
from datetime import datetime
from dotenv import load_dotenv

# Load environment variables
load_dotenv()

class WeatherDashboard:
    def __init__(self):
        self.api_key = os.getenv('OPENWEATHER_API_KEY')
        self.bucket_name = os.getenv('AWS_BUCKET_NAME')
        self.s3_client = boto3.client('s3')

    def create_bucket_if_not_exists(self):
        """Create S3 bucket if it doesn't exist"""
        try:
            self.s3_client.head_bucket(Bucket=self.bucket_name)
            print(f"Bucket {self.bucket_name} exists")
        except:
            print(f"Creating bucket {self.bucket_name}")
        try:
            # Simpler creation for us-east-1
            self.s3_client.create_bucket(Bucket=self.bucket_name)
            print(f"Successfully created bucket {self.bucket_name}")
        except Exception as e:
            print(f"Error creating bucket: {e}")

    def fetch_weather(self, city):
        """Fetch weather data from OpenWeather API"""
        base_url = "http://api.openweathermap.org/data/2.5/weather"
        params = {
            "q": city,
            "appid": self.api_key,
            "units": "imperial"
        }

        try:
            response = requests.get(base_url, params=params)
            response.raise_for_status()
            return response.json()
        except requests.exceptions.RequestException as e:
            print(f"Error fetching weather data: {e}")
            return None

    def save_to_s3(self, weather_data, city):
        """Save weather data to S3 bucket"""
        if not weather_data:
            return False

        timestamp = datetime.now().strftime('%Y%m%d-%H%M%S')
        file_name = f"weather-data/{city}-{timestamp}.json"

        try:
            weather_data['timestamp'] = timestamp
            self.s3_client.put_object(
                Bucket=self.bucket_name,
                Key=file_name,
                Body=json.dumps(weather_data),
                ContentType='application/json'
            )
            print(f"Successfully saved data for {city} to S3")
            return True
        except Exception as e:
            print(f"Error saving to S3: {e}")
            return False

def main():
    dashboard = WeatherDashboard()

    # Create bucket if needed
    dashboard.create_bucket_if_not_exists()

    cities = ["Philadelphia", "Seattle", "New York"]

    for city in cities:
        print(f"\nFetching weather for {city}...")
        weather_data = dashboard.fetch_weather(city)
        if weather_data:
            temp = weather_data['main']['temp']
            feels_like = weather_data['main']['feels_like']
            humidity = weather_data['main']['humidity']
            description = weather_data['weather'][0]['description']

            print(f"Temperature: {temp}°F")
            print(f"Feels like: {feels_like}°F")
            print(f"Humidity: {humidity}%")
            print(f"Conditions: {description}")

            # Save to S3
            success = dashboard.save_to_s3(weather_data, city)
            if success:
                print(f"Weather data for {city} saved to S3!")
        else:
            print(f"Failed to fetch weather data for {city}")

if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

This part of the code imports the packages used and loads the environment variables from the .env file.

  • os package allows interaction with the operating system (e.g file paths)
  • json package allows easy parsing of json objects
  • boto3 package allows easy interaction with AWS resources
  • requests package allows making https requests
  • datetime package for working with date and time
  • dotenv package for loading the environment variables from the .env file.
import os
import json
import boto3
import requests
from datetime import datetime
from dotenv import load_dotenv

# Load environment variables
load_dotenv()
Enter fullscreen mode Exit fullscreen mode

This part of the code initializes the api_key, bucket_name, and s3_client properties of the WeatherDashboard class.

class WeatherDashboard:
    def __init__(self):
        self.api_key = os.getenv('OPENWEATHER_API_KEY')
        self.bucket_name = os.getenv('AWS_BUCKET_NAME')
        self.s3_client = boto3.client('s3')
Enter fullscreen mode Exit fullscreen mode

This method checks if the bucket exists and if not it proceeds to creating the s3 bucket.

    def create_bucket_if_not_exists(self):
        """Create S3 bucket if it doesn't exist"""
        try:
            self.s3_client.head_bucket(Bucket=self.bucket_name)
            print(f"Bucket {self.bucket_name} exists")
        except:
            print(f"Creating bucket {self.bucket_name}")
        try:
            # Simpler creation for us-east-1
            self.s3_client.create_bucket(Bucket=self.bucket_name)
            print(f"Successfully created bucket {self.bucket_name}")
        except Exception as e:
            print(f"Error creating bucket: {e}")
Enter fullscreen mode Exit fullscreen mode

This method fetches the weather data by for the specified city and returns the response data in json format or None (in case of error).

    def fetch_weather(self, city):
        """Fetch weather data from OpenWeather API"""
        base_url = "http://api.openweathermap.org/data/2.5/weather"
        params = {
            "q": city,
            "appid": self.api_key,
            "units": "imperial"
        }

        try:
            response = requests.get(base_url, params=params)
            response.raise_for_status()
            return response.json()
        except requests.exceptions.RequestException as e:
            print(f"Error fetching weather data: {e}")
            return None
Enter fullscreen mode Exit fullscreen mode

This method saves the weather data to s3. It creates a unique json file ({city}-{timestamp}.json) and saves it in the weather-data folder on S3.

    def save_to_s3(self, weather_data, city):
        """Save weather data to S3 bucket"""
        if not weather_data:
            return False

        timestamp = datetime.now().strftime('%Y%m%d-%H%M%S')
        file_name = f"weather-data/{city}-{timestamp}.json"

        try:
            weather_data['timestamp'] = timestamp
            self.s3_client.put_object(
                Bucket=self.bucket_name,
                Key=file_name,
                Body=json.dumps(weather_data),
                ContentType='application/json'
            )
            print(f"Successfully saved data for {city} to S3")
            return True
        except Exception as e:
            print(f"Error saving to S3: {e}")
            return False
Enter fullscreen mode Exit fullscreen mode

The main function is where we do all the magic. We initialize the dashboard instance using the WeatherDashboard class. We then create the S3 bucket if it doesn't exist. After which we iterate through our cities, fetch the weather data for each city, then extract the temparature, feel_like, humidity, and description data and print it to the console. Finally we save the weather data into s3.

def main():
    dashboard = WeatherDashboard()

    # Create bucket if needed
    dashboard.create_bucket_if_not_exists()

    cities = ["Philadelphia", "Seattle", "New York"]

    for city in cities:
        print(f"\nFetching weather for {city}...")
        weather_data = dashboard.fetch_weather(city)
        if weather_data:
            temp = weather_data['main']['temp']
            feels_like = weather_data['main']['feels_like']
            humidity = weather_data['main']['humidity']
            description = weather_data['weather'][0]['description']

            print(f"Temperature: {temp}°F")
            print(f"Feels like: {feels_like}°F")
            print(f"Humidity: {humidity}%")
            print(f"Conditions: {description}")

            # Save to S3
            success = dashboard.save_to_s3(weather_data, city)
            if success:
                print(f"Weather data for {city} saved to S3!")
        else:
            print(f"Failed to fetch weather data for {city}")
Enter fullscreen mode Exit fullscreen mode

Finally, This code ensures the main function runs when the file is directly executed.

if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Step 9

Run the code.

python3 src/weather_dashboard.py
Enter fullscreen mode Exit fullscreen mode

If everything goes right, you should see this on your terminal.

script-result

Navigate to your S3 console and check out the bucket.

weather bucket

Click on the weather-data bucket to see the weather data
weather data

To view the JSON data, click on any of the city data and then click open
open json data

The JSON data looks like this
city json data

Voila! we are done.

Step 10

It's good practice to clean up after you're done. So you can navigate to your S3 console, empty the bucket then proceed to delete.

Cleanup data
Conclusion

In this project, we created an S3 bucket using the boto3 AWS SDK, we then loaded weather data into the S3 bucket. Follow me for more cloud-based projects.

Top comments (0)