Weather Dashboard using AWS and Python

Introduction!

This project is a part of day 1 of 30 DevOps challenge! The projects uses AWS Lambda, SSM parameter store, python, AWS S3 and Quicksight.

The first part of the project can be run using a local setup without using Lambda and the details for it can be found on github

This article focuses on implementation using AWS lambda, SSM parameter store and QuickSight for visualization!

Architecture Digram

The lambda function uses boto3 to create S3 buckets and upload data, SSM parameter store is used to store API key to ensure it is not exposed, the bucket names are environment variables making it easy to facilitate any changes if needed. QuickSight is used to visualize data. The local machine set up is for anyone wanting to run the python script locally!

Implementation

Step 1:

Clone the repository locally which can be done using
git clone https://github.com/anuj672/weather-dashboard

Step 2 - Setting up The lambda function

• Create a new lambda function in AWS using Python as the run type.
• In the Github folder copy code under the ssrc_lambda/weather_dashboard.py directory and paste it in the function.
• From the UI navigate to the configuration tab in your lambda function and add two new environment variables, these will be your bucket names to store json and csv files respectively.

AWS_BUCKET_NAME=<your bucket name>
AWS_VISUALIZE_BUCKET_NAME=<your bucket name>

• Increase the timeout of the function to 15 seconds from general configuration -> edit

Step 3 - Create a parameter for the API inside the SSM parameter store

Inside AWS console, search for "parameter store" , and and click on create new parameter. Create a new parameter of the name
OPENWEATHER_API_KEY and choose the type as secureString. Give it a value of your API key generated from the openweather website.

Step 4 - Modifying the IAM role of the lambda function to allow access to S3 and Parameter store.

• To allow access to lambda function to the Secret parameter used inside the script and also to allow access to S3 we can either create an inline policy or edit the existing policy of the lambda function.

• Under the configuration tab inside the lambda function, click on permissions tab and click on the role name.

• Under IAM select the option to add inline policies

• select the service as system manager and give the following read permissions and save the policy ssm:GetParametersByPath, ssm:GetParameters, ssm:GetParameter

permissions needed

• You can follow a similar approach for S3 or attach a policy as well

The below policy can be excessively permissive as it applies to all buckets but allows us to give any bucket name as a part of the environment variables.

        {
            "Effect": "Allow",
            "Action": [
                "s3:CreateBucket",
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutBucketPolicy"
            ],
            "Resource": "arn:aws:s3:::*"
        }

Step 5 - Running the Lambda Function!

• Once this setup is complete we can now run our lambda function.
• Create a test event with inputs as follows

{
  "cities": [
    "Philadelphia",
    "Texas",
    "New York"
  ]
}

• Deploy and run the code
• If done correctly it will create two s3 buckets (if they don't exist) and return 200!

Step 6 - Integrating our S3 bucket (with csv data) to Amazon QuickSight

• Create a new Amazon Quicksight account from here

• Navigate to the S3 bucket and upload a manifest.json file from the src_lambda folder in the Github repository to the root directory

• Copy the S3 url, allow QuickSight to access it and set up a analysis in QuickSight by giving the url to the manifest file

• Demo plot! from QuickSight!!

Thank you and Way forward!

• This has truly been a great project packed with learning! Special thanks to the cozycloudcrew for giving me this opportunity.
• Collaborate with me? List on open issues which can be done to take this forward can be found here.

Resources and credits

1. https://docs.aws.amazon.com/lambda/latest/dg/python-package.html
2. https://docs.aws.amazon.com/quicksight/latest/user/supported-manifest-file-format.html
3. https://youtu.be/A95XBJFOqjw?si=sQUKWmV82B0w7-_8
4. https://github.com/techwithlucy/youtube/tree/main/2-s3-quicksight
5. AWS documents for navigating some policy related changes