IMDb-MongoDb

Adhir Kirtikar — Wed, 12 Jan 2022 08:38:36 +0000

Overview of My Submission

This project is created for participating in MongoDB Atlas Hackathon on DEV in category "Choose Your Own Adventure" and to learn more about the modern database platform of MongoDB Atlas.

This is a simple Node.js Express web app in which you can search for movies based on various criteria and see the stats & details of the movies e.g. duration, language, ratings, Cast & Crew etc.
I have not learnt Node.js Express or pug before. I created a new project in Visual Studio, experimented with the base code created by VS and modified the code as required based on my requirements by googling for solutions.

This is a Node.js Express web app with only 2 routes: index & search.
The source code is in this repo IMDb-MongoDb.
The code uses Node.js native drivers for MongoDB and connects to a MongoDB database deployment which is Multi Region Replica Set of 12 nodes.
The database consists of data from IMDb sourced from a Kaggle dataset.
The app is packaged in a Docker container automagically using GitHub actions.
The docker container is deployed to 3-node Kubernetes cluster hosted on Civo using Helm charts with ArgoCD which was installed very easily and quickly in the Civo cluster.
The app endpoint is published via Traefik Ingress (again auto install in Civo cluster) and is embedded in the home page hosted on .xyz domain. (Note: There were issues with embedding the application link with the Weebly homepage editor, so the direct link for the application is as below. The xyz link gives "Bad Gateway" error some times.)
The webpage link for the application is IMDb-MongoDb.

Submission Category: Choose Your Own Adventure

I only wanted to learn MongoDB and participate in this challenge. The idea that I had for movies search app does not fit in to the other categories.

Link to Code

AdhirKirtikar / IMDb-MongoDb

IMDb movies web app for MongoDB Atlas Hackathon on DEV!

IMDb-MongoDb

Inspiration

This project is created for participating in MongoDB Atlas Hackathon on DEV in category "Choose Your Own Adventure" and to learn more about the modern database platform of MongoDB Atlas.

What it does

This is a simple Node.js Express web app in which you can search for movies based on various criteria and see the stats & details of the movies e.g. duration, language, ratings, Cast & Crew etc.

How I built it

This is a Node.js Express web app with only 2 routes: index & search.
The source code is in this repo IMDb-MongoDb.
The code uses Node.js native drivers for MongoDB and connects to a MongoDB database deployment which is Multi Region Replica Set of 12 nodes. (Update 20220124: Changed to MongoDB Free Tier)
The database consists of data from imdb sourced from a Kaggle dataset.
The app is packaged in a Docker…

View on GitHub

Additional Resources / Info

Video Demo: MongoDB Atlas Hackathon on DEV - iMovieDB

Air Quality - Pollutant Index - India

Adhir Kirtikar — Wed, 24 Nov 2021 18:40:28 +0000

This mini project shows the Air Quality Index (Industrial Air Pollution) from various locations in India in a Tableau Public dashboard.
The data is sourced from data.gov.in API using Python for cleaning & loading the data in Google Sheets.
Data is updated daily (or manually) using GitHub Actions or AWS Lambda.

My Workflow

GitHub Action "run-python.yml"

Google credentials are stored in GitHub Actions Environment Secrets.
data.gov.in API Key is also stored in GitHub Actions Environment Secrets.
A GitHub Action is created that can run manually or on a schedule [12PM UTC (8PM SGT)].
Python 3.9 is setup using actions/setup-python@v2.3.0 and the pip packages are cached.
Dependencies are installed using py-actions/py-dependency-install@v2 based on requirements.txt (google auth, pygsheets & pandas).
The Environment Secrets are exported to environment variables.
Finally, the Python script is run with the environment variables passed as parameters.

Submission Category:

Wacky Wildcards

I tried to use this workflow as a replacement / complement to the AWS Lambda function that processes the Python script at 8AM SGT.

Yaml File or Link to Code

run-python.yml

# This is a basic workflow to help you get started with Actions

name: Run Python

# Controls when the action will run. 
on:
  schedule:
    # run at 12PM UTC (8PM SGT)
    - cron: '0 12 * * *'

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # use environment (named as "env") defined in the GitHub repository settings
    environment: env

    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      -
        name: Checkout
        uses: actions/checkout@v2

      # Set up Python 3.9 environment and cache pip packages
      - 
        name: Setup Python 3.9
        uses: actions/setup-python@v2.3.0
        with:
          python-version: '3.9'
          cache: 'pip'
        # This action sets up a Python environment for use in actions by:
        #   optionally installing and adding to PATH a version of Python that is already installed in the tools cache.
        #   optionally caching dependencies for pip and pipenv.

      # Install dependencies mentioned in the requirements.txt
      - 
        name: Install dependencies
        uses: py-actions/py-dependency-install@v2
        # This GitHub Action installs Python package dependencies from a user-defined requirements.txt file path 
        # with pip, setuptools, and wheel installs/updates during execution. 
        # A Python package environment report is displayed at the end of Action execution.
        # Uses path requirements.txt and updates pip, setuptools, and wheel before the install.

      # Run a bash shell and store env secrets in parameters to pass to Python script
      -
        name: Get Parameters & Run Air Quality Index India Python script
        shell: bash
        env:
          GOVINAPIKEY: ${{ secrets.DATA_GOV_IN_API_KEY }}
          GDRIVEAPIKEY: ${{ secrets.GDRIVE_API_CREDENTIALS }}
        run: |
          python "Air Quality Index India.py" "$GOVINAPIKEY" "$GDRIVEAPIKEY"

Full repository is here:

AdhirKirtikar / Air-Quality-Index-India-GitHub-Actions

Repo for 2021 GitHub Actions Hackathon on DEV

Air-Quality-Index-India-GitHub-Actions

Repo for 2021 GitHub Actions Hackathon on DEV

GitHub Action "run-python.yml"

Google credentials are stored in GitHub Actions Environment Secrets.
data.gov.in API Key is also stored in GitHub Actions Environment Secrets.
A GitHub Action is created that can run manually or on a schedule [12PM UTC (8PM SGT)]
Python 3.9 is setup using actions/setup-python@v2.3.0 and the pip packages are cached
Dependencies are installed using py-actions/py-dependency-install@v2 based on requirements.txt (google auth, pygsheets & pandas).
The Environment Secrets are exported to environment variables.
Finally, the Python script is run with the environment variables passed as parameters.

Python script "Air Quality Index India.py"

The script connects to data.gov.in API using API Key passed as a parameter.
Then it pulls the latest AQI data for India and stores in pandas dataframe.
The data is cleaned, formatted and the columns are renamed. Nulls are replaced by 0.
Google sheet is…

View on GitHub

Additional Resources / Info

Tableau Public Dashboard that uses the data from the generated Google Sheets: "Air Quality - Pollutant Index - India"

DEV Community: Adhir Kirtikar

IMDb-MongoDb

Overview of My Submission

Submission Category: Choose Your Own Adventure

Link to Code

AdhirKirtikar / IMDb-MongoDb

IMDb movies web app for MongoDB Atlas Hackathon on DEV!

IMDb-MongoDb

Inspiration

What it does

How I built it

Additional Resources / Info

Air Quality - Pollutant Index - India

My Workflow

GitHub Action "run-python.yml"

Submission Category:

Yaml File or Link to Code

run-python.yml

Full repository is here:

AdhirKirtikar / Air-Quality-Index-India-GitHub-Actions

Repo for 2021 GitHub Actions Hackathon on DEV

Air-Quality-Index-India-GitHub-Actions

Repo for 2021 GitHub Actions Hackathon on DEV

GitHub Action "run-python.yml"

Python script "Air Quality Index India.py"

Additional Resources / Info