In this article, we will look at how to store JSON data in an Amazon S3 bucket.
Technologies & Tools
- Python - Run time environment
- AWS Account (Free Tier)
- Boto3 - AWS SDK for python
-
python-dotenv - to load environment variables from
.env
files. - requests - a simple HTTPS library
- Openweather API - external api that provides real time weather data
- Github Repo (Optional) - a version control for your code
Pre-requisite
Setup Python
if you haven't, please setup Python on you local machine.
[Optional] Github Account & Repo
Skip this if you don't intend to save your project on github. However, you can follow these links to setup a Github Account & Repo.
Setup AWS account and IAM User
Create you AWS Free Tier account if you haven't. Then, follow these steps to create your IAM user. Ensure the IAM user is assigned the AmazonS3FullAccess
policy.
Setup AWS CLI
We are going to be using the Boto3 AWS SDK to interact with AWS. Underneath, the AWS uses a AWS API KEY and secret key to authenticate with AWS. Hence we need to configure AWS credentials on our local machine. Please follow the Boto3 documentation for configuration guide. Ensure the IAM user credentials you're using is configured with the AmazonS3FullAccess
policy mentioned above.
OpenWeather API
Openweather is the external api that we will use in this project to provide real time weather data. Please sign up to OpenWeather service to get your OpenWeather API key.
Now let's get started! 😀
P.S the CLI commands examples will be for MacOs/linux users. For windows users you can research the commands online.
Step 1
Create a folder called weather-dashboard-demo
mkdir weather-dashboard-demo
Step 2
Change directory to weather-dashboard-demo and create necessary files and folders
cd weather-dashboard-demo
mkdir src
touch .env requirements.txt
Step 3
Initialize git (optional)
git init
Step 4
Add the package dependencies in re requirements.txt
echo "boto3==1.26" >> requirements.txt
echo "python-dotenv==1.0.0" >> requirements.txt
echo "requests==2.28" >> requirements.txt
Step 5
Install your dependencies
pip3 install -r requirements.txt
Step 6
Add env variables to your .env file.
Get your API key from the open weather account you created above.
echo "OPENWEATHER_API_KEY=<replace-with-your-api-key>" >> .env
Add your s3 bucket name. Note your bucket name must be globally unique.
echo "AWS_BUCKET_NAME=weather-dashboard-<append-unique-value>" >> .env
Step 7
Create the necessary files in the src folder
touch src/__init__.py src/weather_dashboard.py
Step 8
Copy the full code into the weather_dashboard.py. Don't feel overwhelmed I will explain the code.
import os
import json
import boto3
import requests
from datetime import datetime
from dotenv import load_dotenv
# Load environment variables
load_dotenv()
class WeatherDashboard:
def __init__(self):
self.api_key = os.getenv('OPENWEATHER_API_KEY')
self.bucket_name = os.getenv('AWS_BUCKET_NAME')
self.s3_client = boto3.client('s3')
def create_bucket_if_not_exists(self):
"""Create S3 bucket if it doesn't exist"""
try:
self.s3_client.head_bucket(Bucket=self.bucket_name)
print(f"Bucket {self.bucket_name} exists")
except:
print(f"Creating bucket {self.bucket_name}")
try:
# Simpler creation for us-east-1
self.s3_client.create_bucket(Bucket=self.bucket_name)
print(f"Successfully created bucket {self.bucket_name}")
except Exception as e:
print(f"Error creating bucket: {e}")
def fetch_weather(self, city):
"""Fetch weather data from OpenWeather API"""
base_url = "http://api.openweathermap.org/data/2.5/weather"
params = {
"q": city,
"appid": self.api_key,
"units": "imperial"
}
try:
response = requests.get(base_url, params=params)
response.raise_for_status()
return response.json()
except requests.exceptions.RequestException as e:
print(f"Error fetching weather data: {e}")
return None
def save_to_s3(self, weather_data, city):
"""Save weather data to S3 bucket"""
if not weather_data:
return False
timestamp = datetime.now().strftime('%Y%m%d-%H%M%S')
file_name = f"weather-data/{city}-{timestamp}.json"
try:
weather_data['timestamp'] = timestamp
self.s3_client.put_object(
Bucket=self.bucket_name,
Key=file_name,
Body=json.dumps(weather_data),
ContentType='application/json'
)
print(f"Successfully saved data for {city} to S3")
return True
except Exception as e:
print(f"Error saving to S3: {e}")
return False
def main():
dashboard = WeatherDashboard()
# Create bucket if needed
dashboard.create_bucket_if_not_exists()
cities = ["Philadelphia", "Seattle", "New York"]
for city in cities:
print(f"\nFetching weather for {city}...")
weather_data = dashboard.fetch_weather(city)
if weather_data:
temp = weather_data['main']['temp']
feels_like = weather_data['main']['feels_like']
humidity = weather_data['main']['humidity']
description = weather_data['weather'][0]['description']
print(f"Temperature: {temp}°F")
print(f"Feels like: {feels_like}°F")
print(f"Humidity: {humidity}%")
print(f"Conditions: {description}")
# Save to S3
success = dashboard.save_to_s3(weather_data, city)
if success:
print(f"Weather data for {city} saved to S3!")
else:
print(f"Failed to fetch weather data for {city}")
if __name__ == "__main__":
main()
This part of the code imports the packages used and loads the environment variables from the .env file.
-
os
package allows interaction with the operating system (e.g file paths) -
json
package allows easy parsing of json objects -
boto3
package allows easy interaction with AWS resources -
requests
package allows making https requests -
datetime
package for working with date and time -
dotenv
package for loading the environment variables from the .env file.
import os
import json
import boto3
import requests
from datetime import datetime
from dotenv import load_dotenv
# Load environment variables
load_dotenv()
This part of the code initializes the api_key, bucket_name, and s3_client properties of the WeatherDashboard class
.
class WeatherDashboard:
def __init__(self):
self.api_key = os.getenv('OPENWEATHER_API_KEY')
self.bucket_name = os.getenv('AWS_BUCKET_NAME')
self.s3_client = boto3.client('s3')
This method checks if the bucket exists and if not it proceeds to creating the s3 bucket.
def create_bucket_if_not_exists(self):
"""Create S3 bucket if it doesn't exist"""
try:
self.s3_client.head_bucket(Bucket=self.bucket_name)
print(f"Bucket {self.bucket_name} exists")
except:
print(f"Creating bucket {self.bucket_name}")
try:
# Simpler creation for us-east-1
self.s3_client.create_bucket(Bucket=self.bucket_name)
print(f"Successfully created bucket {self.bucket_name}")
except Exception as e:
print(f"Error creating bucket: {e}")
This method fetches the weather data by for the specified city and returns the response data in json format or None (in case of error).
def fetch_weather(self, city):
"""Fetch weather data from OpenWeather API"""
base_url = "http://api.openweathermap.org/data/2.5/weather"
params = {
"q": city,
"appid": self.api_key,
"units": "imperial"
}
try:
response = requests.get(base_url, params=params)
response.raise_for_status()
return response.json()
except requests.exceptions.RequestException as e:
print(f"Error fetching weather data: {e}")
return None
This method saves the weather data to s3. It creates a unique json file ({city}-{timestamp}.json) and saves it in the weather-data folder on S3.
def save_to_s3(self, weather_data, city):
"""Save weather data to S3 bucket"""
if not weather_data:
return False
timestamp = datetime.now().strftime('%Y%m%d-%H%M%S')
file_name = f"weather-data/{city}-{timestamp}.json"
try:
weather_data['timestamp'] = timestamp
self.s3_client.put_object(
Bucket=self.bucket_name,
Key=file_name,
Body=json.dumps(weather_data),
ContentType='application/json'
)
print(f"Successfully saved data for {city} to S3")
return True
except Exception as e:
print(f"Error saving to S3: {e}")
return False
The main
function is where we do all the magic. We initialize the dashboard instance using the WeatherDashboard class. We then create the S3 bucket if it doesn't exist. After which we iterate through our cities, fetch the weather data for each city, then extract the temparature, feel_like, humidity, and description data and print it to the console. Finally we save the weather data into s3.
def main():
dashboard = WeatherDashboard()
# Create bucket if needed
dashboard.create_bucket_if_not_exists()
cities = ["Philadelphia", "Seattle", "New York"]
for city in cities:
print(f"\nFetching weather for {city}...")
weather_data = dashboard.fetch_weather(city)
if weather_data:
temp = weather_data['main']['temp']
feels_like = weather_data['main']['feels_like']
humidity = weather_data['main']['humidity']
description = weather_data['weather'][0]['description']
print(f"Temperature: {temp}°F")
print(f"Feels like: {feels_like}°F")
print(f"Humidity: {humidity}%")
print(f"Conditions: {description}")
# Save to S3
success = dashboard.save_to_s3(weather_data, city)
if success:
print(f"Weather data for {city} saved to S3!")
else:
print(f"Failed to fetch weather data for {city}")
Finally, This code ensures the main function runs when the file is directly executed.
if __name__ == "__main__":
main()
Step 9
Run the code.
python3 src/weather_dashboard.py
If everything goes right, you should see this on your terminal.
Navigate to your S3 console and check out the bucket.
Click on the weather-data
bucket to see the weather data
To view the JSON data, click on any of the city data and then click open
Voila! we are done.
Step 10
It's good practice to clean up after you're done. So you can navigate to your S3 console, empty the bucket then proceed to delete.
In this project, we created an S3 bucket using the boto3 AWS SDK, we then loaded weather data into the S3 bucket. Follow me for more cloud-based projects.
Top comments (0)