How To Get an API Data and Store in AWS S3

Data is used in nearly every industry, and one of the easiest ways to get it is through APIs.

In this tutorial, we’ll show you how to get data from the OpenWeather API using Python and save it in AWS S3. This is a simple way to fetch data from an API and store it in the cloud for later use. If you’re new to this, don’t worry—we’ll walk you through the steps.

If you’re not familiar with Python, you can check out this article on How to Fetch API Data using React to learn a different approach.

What You’ll Learn

By the end of this tutorial, you will know how to:

• Fetch weather data from the OpenWeather API using Python.
• Set up an S3 bucket to store the data.
• Upload the fetched data to AWS S3.

Prerequisites

Before you start, make sure you have:

• An AWS account. If you don't have one, sign up here.

• A GitHub repository with your source code. If you don’t have a GitHub account,sign up here.

• A code editor. For this tutorial, I used VS Code.

Step 1: Create an AWS S3 Bucket

To store your data, you'll need to create an S3 bucket in your AWS account. Follow these steps:

• Log in to your AWS account.

• In the search bar, type S3.

• Click on Create bucket and follow the on-screen instructions.

• Choose a unique name for your bucket (e.g., my-weather-data-bucket).

• Select your preferred region.

• Click Create to finish.

Step 2: Fetch Data from OpenWeather API

Sign up on OpenWeather.

How to Get Your API Key

• Sign Up: First, sign up for an account on the OpenWeather website. After signing up, you will be redirected to a new page.

• Find the API Key:

  • On the top tab of the page, look for a section labelled API Key.
  • Click on the API Key tab to view your key.

• OR, Access Through Your Profile:
  • Go to your Profile Name section.
  • Click on it, and a drop-down menu will appear with the option to view your API key.

• Install required libraries:

• Install the requests library to make API requests:
  

pip install requests

• Fetch weather data:

import requests
import json

api_key = 'YOUR_API_KEY'  # Replace with your OpenWeather API key
city = 'London'  # You can change this to any city you like

def fetch_weather_data():
    url = f'http://api.openweathermap.org/data/2.5/weather?q={city}&appid={api_key}'
    response = requests.get(url)
    data = response.json()
    return data

weather_data = fetch_weather_data()
print(weather_data)

In this script, we send a request to the OpenWeather API and retrieve the current weather data for the specified city.

Step 3: Set Up AWS SDK for Python (Boto3)

To interact with AWS services, we’ll need the Boto3 library, which is the AWS SDK for Python.

• Install Boto3:

pip install boto3

• Configure AWS credentials:

Set up your AWS access credentials by following the configuration guide.
You’ll need your AWS Access Key and AWS Secret Key.

Step 4: Upload Data to AWS S3

Now, we can upload the weather data to AWS S3.

Set up the S3 client in Python:

import boto3

# AWS credentials and region
aws_access_key_id = 'YOUR_ACCESS_KEY'  # Replace with your AWS Access Key
aws_secret_access_key = 'YOUR_SECRET_KEY'  # Replace with your AWS Secret Key
region_name = 'eu-west-2'  # Replace with your preferred region

# Initialize the S3 client
s3 = boto3.client(
    's3',
    aws_access_key_id=aws_access_key_id,
    aws_secret_access_key=aws_secret_access_key,
    region_name=region_name
)

• Upload the weather data:

def upload_data_to_s3(data):
    bucket_name = 'my-weather-data-bucket'  # Replace with your bucket name
    file_name = 'weather-data.json'  # File name in S3
    s3.put_object(
        Bucket=bucket_name,
        Key=file_name,
        Body=json.dumps(data),
        ContentType='application/json'
    )
    print('Data uploaded successfully to S3')

upload_data_to_s3(weather_data)

How It Works:

• Fetching Data: We use Python’s requests library to send a GET request to the OpenWeather API and get the weather data in JSON format.

• Uploading to S3: We use the boto3 library to upload the fetched JSON data to the S3 bucket as a file named weather-data.json.

Step 5: Verify the Upload

Once you’ve run the script, go to the S3 Management Console in AWS. You should be able to see the file weather-data inside your S3 bucket.

Conclusion

In this tutorial, you’ve learned how to fetch weather data from the OpenWeather API using Python and upload it to AWS S3 for storage. This is a useful process if you need to store and access API data in the cloud for future use.