DEV Community

Cover image for Air Quality - Pollutant Index - India
Adhir Kirtikar
Adhir Kirtikar

Posted on • Edited on

4 1

Air Quality - Pollutant Index - India

This mini project shows the Air Quality Index (Industrial Air Pollution) from various locations in India in a Tableau Public dashboard.
The data is sourced from data.gov.in API using Python for cleaning & loading the data in Google Sheets.
Data is updated daily (or manually) using GitHub Actions or AWS Lambda.

My Workflow

GitHub Action "run-python.yml"

  • Google credentials are stored in GitHub Actions Environment Secrets.
  • data.gov.in API Key is also stored in GitHub Actions Environment Secrets.
  • A GitHub Action is created that can run manually or on a schedule [12PM UTC (8PM SGT)].
  • Python 3.9 is setup using actions/setup-python@v2.3.0 and the pip packages are cached.
  • Dependencies are installed using py-actions/py-dependency-install@v2 based on requirements.txt (google auth, pygsheets & pandas).
  • The Environment Secrets are exported to environment variables.
  • Finally, the Python script is run with the environment variables passed as parameters.

Submission Category:

Wacky Wildcards

  • I tried to use this workflow as a replacement / complement to the AWS Lambda function that processes the Python script at 8AM SGT.

Yaml File or Link to Code

run-python.yml

# This is a basic workflow to help you get started with Actions

name: Run Python

# Controls when the action will run. 
on:
  schedule:
    # run at 12PM UTC (8PM SGT)
    - cron: '0 12 * * *'

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # use environment (named as "env") defined in the GitHub repository settings
    environment: env

    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      -
        name: Checkout
        uses: actions/checkout@v2

      # Set up Python 3.9 environment and cache pip packages
      - 
        name: Setup Python 3.9
        uses: actions/setup-python@v2.3.0
        with:
          python-version: '3.9'
          cache: 'pip'
        # This action sets up a Python environment for use in actions by:
        #   optionally installing and adding to PATH a version of Python that is already installed in the tools cache.
        #   optionally caching dependencies for pip and pipenv.

      # Install dependencies mentioned in the requirements.txt
      - 
        name: Install dependencies
        uses: py-actions/py-dependency-install@v2
        # This GitHub Action installs Python package dependencies from a user-defined requirements.txt file path 
        # with pip, setuptools, and wheel installs/updates during execution. 
        # A Python package environment report is displayed at the end of Action execution.
        # Uses path requirements.txt and updates pip, setuptools, and wheel before the install.

      # Run a bash shell and store env secrets in parameters to pass to Python script
      -
        name: Get Parameters & Run Air Quality Index India Python script
        shell: bash
        env:
          GOVINAPIKEY: ${{ secrets.DATA_GOV_IN_API_KEY }}
          GDRIVEAPIKEY: ${{ secrets.GDRIVE_API_CREDENTIALS }}
        run: |
          python "Air Quality Index India.py" "$GOVINAPIKEY" "$GDRIVEAPIKEY"
Enter fullscreen mode Exit fullscreen mode

Full repository is here:

GitHub logo AdhirKirtikar / Air-Quality-Index-India-GitHub-Actions

Repo for 2021 GitHub Actions Hackathon on DEV

Air-Quality-Index-India-GitHub-Actions

Repo for 2021 GitHub Actions Hackathon on DEV

GitHub Action "run-python.yml"

  • Google credentials are stored in GitHub Actions Environment Secrets.
  • data.gov.in API Key is also stored in GitHub Actions Environment Secrets.
  • A GitHub Action is created that can run manually or on a schedule [12PM UTC (8PM SGT)]
  • Python 3.9 is setup using actions/setup-python@v2.3.0 and the pip packages are cached
  • Dependencies are installed using py-actions/py-dependency-install@v2 based on requirements.txt (google auth, pygsheets & pandas).
  • The Environment Secrets are exported to environment variables.
  • Finally, the Python script is run with the environment variables passed as parameters.

Python script "Air Quality Index India.py"

  • The script connects to data.gov.in API using API Key passed as a parameter.
  • Then it pulls the latest AQI data for India and stores in pandas dataframe.
  • The data is cleaned, formatted and the columns are renamed. Nulls are replaced by 0.
  • Google sheet is…

Additional Resources / Info

Tableau Public Dashboard that uses the data from the generated Google Sheets: "Air Quality - Pollutant Index - India"

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more

👋 Kindness is contagious

Explore a sea of insights with this enlightening post, highly esteemed within the nurturing DEV Community. Coders of all stripes are invited to participate and contribute to our shared knowledge.

Expressing gratitude with a simple "thank you" can make a big impact. Leave your thanks in the comments!

On DEV, exchanging ideas smooths our way and strengthens our community bonds. Found this useful? A quick note of thanks to the author can mean a lot.

Okay