DEV Community

peter279k
peter279k

Posted on

Introduce to my Taiwan COVID-19 news website and crawler

Instructions

In Taiwan, it's COVID-19 outbreak on May 13 and there're about 100 local cases every day. And it's zero local case since October.

At that moment, I'm really concerned about daily local cases so I develop this project and publish it to the GitHub. I use the cron job feature to fetch the latest COVID-19 news and publish my new web page during GitHub action running.

My Workflow

My completed workflow files are available here and it includes following YAML files:

  • fetcher.yml, this YAML file is for fetching latest COVID-19 news and store them to be JSON file. This web crawler is written in Python. And It will be run every hour.
  • web_build.yml, this YAML file is for publishing new website and loading new JSON file. This website is written in JavaScript with Vue.js framework. And it will be run every seventy minutes.

Submission Category:

This category is for DIY-Deployments and I hope developers love this and can be referenced :-).

YAML files and link to repository

Here are repository preview and completed codes about above two YAML files:


  • fetcher.yml
name: Update CSV Dataset Records from Taiwan CDC website

on:
  schedule:
    - cron: '0 * * * *'
  workflow_dispatch:

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Get working copy
        uses: actions/checkout@master
        with:
          fetch-depth: 1
      - name: Set up Python 3.8
        uses: actions/setup-python@v2
        with:
          python-version: '3.8'
      - name: Install Requirements
        run: pip install -r requirements.txt
      - name: Update COVD-19 Labs
        run: python Fetcher/LabFetcher.py
      - name: Update MOHW RSS News
        run: python Fetcher/MohwRssFetcher.py
      - name: Update COVID-19 global cases and deaths
        run: python Fetcher/GlobalCasesFetcher.py
      - name: Update COVID-19 TW datasets
        run: python Fetcher/DailyCovid19TW.py
      - name: Copy JSON files to assets folder
        run: cp datasets/*.json web/src/assets/
      - name: Commit and push if it changed
        run: |
          git config user.name "peter279k"
          git config user.email "peter279k@gmail.com"
          git add -A
          timestamp=$(date -u)
          git commit -m "Last Commit: ${timestamp}(TW)" || exit 0
          git push origin master
Enter fullscreen mode Exit fullscreen mode
  • web_build.yml
name: 'Web Build and Deploy'

on:
  schedule:
    - cron: '10 * * * *'
  workflow_dispatch:
  push:
    branches:
      - main

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout 🛎️
        uses: actions/checkout@v2.3.1
        with:
          persist-credentials: false

      - name: Install and Build 🔧
        working-directory: web
        run: |
          cp .env.example .env
          export VUE_APP_MAPKEY=${{ secrets.VUE_APP_MAPKEY }}
          sed -i -e "s/MAP_KEY/$VUE_APP_MAPKEY/g" ./.env
          npm install
          npm run build
          cd dist
          cp index.html 404.html

      - name: Deploy 🚀
        uses: JamesIves/github-pages-deploy-action@3.6.2
        with:
          GIT_CONFIG_NAME: "peter279k"
          GIT_CONFIG_EMAIL: "peter279k@gmail.com"
          SINGLE_COMMIT: true
          GITHUB_TOKEN: ${{ secrets.TOKEN }}
          BRANCH: gh-pages
          FOLDER: web//dist
Enter fullscreen mode Exit fullscreen mode

Additional Resources / Info

Here are some useful references about GitHub action workflows:

Top comments (0)