DEV Community: peaceiris

Start GitHub Actions self-hosted runner with VirtualBox and Vagrant

peaceiris — Wed, 30 Jun 2021 03:57:17 +0000

Overview

Start GitHub Actions self-hosted runner with VirtualBox and Vagrant on your local machine or a remote machine like Google Compute Engine and EC2.
Avoid dind (Docker in Docker)
Accelerate docker build (or docker-compose build) on the virtual machine with the Docker Layer Caching as the same as your local environment.
Host OS: Ubuntu 18.04/20.04, macOS, Windows
Guest OS: Ubuntu 20.04

Install VirtualBox

Oracle VM VirtualBox

Ubuntu 18.04/20.04



sudo apt install -y virtualbox

macOS



brew cask install virtualbox

Allow Oracle America, Inc. in the System Preferences.

Install Vagrant

hashicorp/vagrant: Vagrant is a tool for building and distributing development environments.

Ubuntu 18.04/20.04



curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt update && sudo apt install -y vagrant
vagrant autocomplete install

macOS



brew install vagrant
vagrant autocomplete install

Start VM



git clone https://github.com/peaceiris/actions-self-hosted-runners.git
cd ./actions-self-hosted-runners/images/ubuntu-20.04
git checkout v0.4.0
vim .env
make up
vagrant ssh
cd actions-runner
./config.sh --url https://github.com/[owner]/[repo] --token [token]
nohup ./run.sh &

Create .env file as follows.



VB_CPUS = '4'
VB_MEMORY = '8192'
VB_DISK_SIZE = '30GB'

VB_CPUS: CPU cores
VB_MEMORY: Memory size
VB_DISK_SIZE: Disk size

Use the VM and Accelerate `docker build`

Unlike the GitHub hosted runner, a self-hosted runner is not destroyed for each job, so Docker Layer Caching will work as well as local machine. You don't have to worry about tricky usage of the actions/cache and capacity limits. The docker build job became accelerated and we achieved the purpose.



name: CI frontend

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  docker:
    runs-on: self-hosted
    timeout-minutes: 3
    env:
      COMPOSE_DOCKER_CLI_BUILD: 1
      DOCKER_BUILDKIT: 1
    steps:
      - uses: actions/checkout@v2.3.4
      - run: hadolint ./Dockerfile
      - run: sudo docker-compose build frontend

Slow docker-compose build job on the GitHub hosted runner.

Accelerated docker-compose build job on a self-hosted GitHub Actions runner.

Enabled Docker Layer Caching on a GitHub Actions self-hosted runner.

Repository

peaceiris / actions-self-hosted-runners

GitHub Actions self-hosted runner on VirtualBox with Vagrant.

actions-self-hosted-runners

GitHub Actions self-hosted runner on VirtualBox with Vagrant.

actions/runner: The Runner for GitHub Actions
actions/virtual-environments: GitHub Actions virtual environments

Getting Started

git clone https://github.com/peaceiris/actions-self-hosted-runners.git
cd ./actions-self-hosted-runners/images/ubuntu
cat << EOF > .env
GHA_RUNNER_VERSION = 'x.xxx.x'
VB_CPUS = '4'
VB_MEMORY = '8192'
VB_DISK_SIZE = '30GB'
EOF
make up
vagrant ssh
cd actions-runner
./config.sh --url https://github.com/[owner]/[repo] --token [token]
nohup ./run.sh &

View on GitHub

Conclusion

We can start a self-hosted GitHub Actions runner on a Ubuntu 20.04 virtual machine with VirtualBox and Vagrant very easily.
We can accelerate docker build (or docker-compose build) on the virtual machine with the Docker Layer Caching as the same as your local environment.

Enjoy the comfortable CI/CD! Thank you.

The Case Sensitivity of the GitHub Actions Runner

peaceiris — Wed, 13 Jan 2021 07:17:53 +0000

The Problem

Bug: Couldn't open SUMMARY.md · Issue #36 · peaceiris/actions-mdbook

mdBook can recognize SUMMARY.md but cannot summary.md on a GitHub Actions runner (Ubuntu VM).

On the other hand, both SUMMARY.md and summary.md are acceptable on my local macOS machine.

The Reason

The case-sensitivity between the GitHub Actions runner and my local environment was different.

My local macOS machine has been formatted with case-insensitive mode, so the local building has passed and the remote building failed.

GitHub Actions Ubuntu: Case sensitive
My local macOS PC: Case insensitive

Can we select case-sensitivity of the GitHub Actions VM?

No.

MacOSX images with case sensitive APFS filesystem · Issue #865 · actions/virtual-environments

We can't offer another image at this time since it would affect overall capacity of Mac and be costly to maintain.

Suggest Related Links Action

peaceiris — Thu, 17 Sep 2020 22:44:50 +0000

My Workflow

Our GitHub Action is peaceiris/actions-suggest-related-links which suggests related or similar issues, documents, and links.

peaceiris / actions-suggest-related-links

A GitHub Action to suggest related or similar issues, documents, and links. Based on the power of NLP and fastText.

Suggest Related Links Action

A GitHub Action to suggest related or similar issues, documents, and links. Based on the power of NLP and fastText.

Sometimes, users create a new issue that is similar to already opened or closed issues. When you face the issue, you need to find similar issues or related links and post it as a comment. That process is a pain, right? This action can do it instead of you!

Getting Started

Create Workflow

Create your YAML workflow file as follows.

e.g. .github/workflows/suggest-related-links.yml

name: 'Suggest Related Links'
on
  issues:
    types:
      - opened
      - edited
  workflow_dispatch:
  schedule:
    - cron: '13 13 * * */7'

jobs:
  action:
    runs-on: ubuntu-18.04
    steps

…

View on GitHub

A GitHub Action to suggest related or similar issues, documents, and links. Based on the power of NLP and fastText.

e.g. .github/workflows/suggest-related-links.yml

name: 'Suggest Related Links'

on:
  issues:
    types:
      - opened
      - edited
  workflow_dispatch:
  schedule:
    - cron: '13 13 * * */7'

jobs:
  action:
    runs-on: ubuntu-18.04
    steps:
      - name: Cache dependencies
        uses: actions/cache@v2
        with:
          path: ~/actions-suggest-related-links-tmp
          key: ${{ runner.os }}-action-${{ hashFiles('~/actions-suggest-related-links-tmp/training-data.json') }}
          restore-keys: |
            ${{ runner.os }}-action-

      - uses: peaceiris/actions-suggest-related-links@v1.1.1
      - uses: peaceiris/actions-suggest-related-links/models/fasttext@v1.1.1
        if: github.event_name == 'issues'
        with:
          version: v1.1.1
      - uses: peaceiris/actions-suggest-related-links@v1.1.1
        with:
          mode: 'suggest'
          repository: 'peaceiris/actions-gh-pages'
          unclickable: true

Workflow Overview

This action mainly consists of 5 parts: Data Collection, Preprocessing, Train model, Find similar issues, and Suggest links.

Data Collection

All issues of the repository are collected with the GitHub API. The issues include the title, body, and comments. Training Data is regularly collected using the scheduling function and output as an artifact and saved as a cache.

Preprocessing

The Markdown format is converted to plain text with unified.
At this time, symbols that are not alphabetic characters are deleted.

https://github.com/peaceiris/actions-suggest-related-links/blob/main/actions/src/preprocess.ts

Train Model

When a new issue is created or updated, the fastText model is trained.
In accordance with its name, fastText has the advantage of very short inference times.

We think training time at the GitHub Actions runners won't be an issue.
In the case of GitHub Actions for GitHub Pages repository, the training execution time is 1 sec and the total execution time is about 30 sec.

Find Similar Issues

Calculate word vectors of training data and word vectors of posted data in fastText. The cosine similarity is used to determine which word vectors of training data is close to the word vectors of the posted data. The higher the cosine similarity, the more similar the sentence.

Suggest Issues

Some related links which are similar to the created issue body will be listed by this action.

https://github.com/peaceiris/actions-suggest-related-links/issues/2

Submission Category

Maintainer Must-Haves

Additional Resources / Info

Maintainers

Dependencies

Deploy to GitHub Pages with GitHub Actions for Static Site Generator

peaceiris — Wed, 11 Sep 2019 04:44:04 +0000

GitHub Actions for deploying to GitHub Pages

Using the following workflow, you can deploy your static site to GitHub Pages with Static Site Generators (Hugo, MkDocs, Gatsby, GitBook, etc.)

- uses: peaceiris/actions-gh-pages@v4
  with:
    github_token: ${{ secrets.GITHUB_TOKEN }}
    publish_dir: ./public

GitHub Action Hero: Shohei Ueda - The GitHub Blog

This GitHub Pages Action, Hugo Action, and mdBook Action were featured at the GitHub Blog. Thanks all!

For more details, go to https://github.com/peaceiris/actions-gh-pages 👇

peaceiris / actions-gh-pages

GitHub Actions for GitHub Pages 🚀 Deploy static files and publish your site easily. Static-Site-Generators-friendly.

GitHub Pages Action

Note

See also the GitHub official GitHub Pages Action first.

GitHub Pages now uses Actions by default | The GitHub Blog

GitHub Pages: Custom GitHub Actions Workflows (beta) | GitHub Changelog

This is a GitHub Action to deploy your static files to GitHub Pages This deploy action can be combined simply and freely with Static Site Generators. (Hugo, MkDocs, Gatsby, mdBook, Next, Nuxt, and so on.)

The next example step will deploy ./public directory to the remote gh-pages branch.

- name: Deploy
  uses: peaceiris/actions-gh-pages@v4
  with:
    github_token: ${{ secrets.GITHUB_TOKEN }}
    publish_dir: ./public

For newbies of GitHub Actions: Note that the GITHUB_TOKEN is NOT a personal access token. A GitHub Actions runner automatically creates a GITHUB_TOKEN secret to authenticate in your workflow. So, you can start to deploy immediately without any configuration.

Supported Tokens

Three tokens are supported.

Token	Private repo	Public

…

View on GitHub

Getting Started

An example YAML file (.github/workflows/gh-pages.yml) with Hugo action.

peaceiris/actions-hugo: GitHub Actions for Hugo

peaceiris / actions-hugo

GitHub Actions for Hugo ⚡️ Setup Hugo quickly and build your site fast. Hugo extended, Hugo Modules, Linux (Ubuntu), macOS, and Windows are supported.

GitHub Actions for Hugo

This Hugo Setup Action can install Hugo to a virtual machine of GitHub Actions Hugo extended version, Hugo Modules, Linux (Ubuntu), macOS, and Windows are supported.

From v2, this Hugo Setup Action has migrated to a JavaScript (TypeScript) action We no longer build or pull a Hugo docker image. Thanks to this change, we can complete this action in less than a few seconds. (A docker base action was taking about 1 min or more execution time to build and pull a docker image.)

OS (runs-on)	ubuntu-latest, ubuntu-20.04, ubuntu-22.04	macos-latest	windows-2019
Support	✅️	✅️	✅️

Hugo type	Hugo Extended	Hugo Modules	Latest Hugo
Support	✅️	✅️	✅️

View on GitHub

name: github pages

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  deploy:
    runs-on: ubuntu-22.04
    concurrency:
      group: ${{ github.workflow }}-${{ github.ref }}
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: true  # Fetch Hugo themes
          fetch-depth: 0    # Fetch all history for .GitInfo and .Lastmod

      - name: Setup Hugo
        uses: peaceiris/actions-hugo@v3
        with:
          hugo-version: '0.85.0'

      - name: Build
        run: hugo --minify

      - name: Deploy
        uses: peaceiris/actions-gh-pages@v4
        if: ${{ github.ref == 'refs/heads/main' }}
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./public

Options

Many options are available! Go to the repository (peaceiris/actions-gh-pages) and check the latest README.

Other Examples

⭐ Static Site Generators with Node.js

next.js, nuxt.js, gatsby, hexo, gitbook, vuepress, react-static, gridsome, etc. (JavaScript projects)

Premise: Dependencies are managed by package.json and package-lock.json

name: github pages

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  deploy:
    runs-on: ubuntu-22.04
    concurrency:
      group: ${{ github.workflow }}-${{ github.ref }}
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '14'

      - name: Cache dependencies
        uses: actions/cache@v4
        with:
          path: ~/.npm
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-node-

      - run: npm ci
      - run: npm run build

      - name: Deploy
        uses: peaceiris/actions-gh-pages@v4
        if: ${{ github.ref == 'refs/heads/main' }}
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./public

⭐️ Static Site Generators with Python

pelican, MkDocs, sphinx, and so on.

Premise: Dependencies are managed by requirements.txt

name: github pages

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  deploy:
    runs-on: ubuntu-22.04
    concurrency:
      group: ${{ github.workflow }}-${{ github.ref }}
    steps:
      - uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.8'
          architecture: 'x64'

      - name: Cache dependencies
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-

      - name: Install dependencies
        run: |
          python3 -m pip install --upgrade pip
          python3 -m pip install -r ./requirements.txt

      - run: mkdocs build

      - name: Deploy
        uses: peaceiris/actions-gh-pages@v4
        if: ${{ github.ref == 'refs/heads/main' }}
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./site

⭐️ mdBook (Rust)

An example GitHub Actions workflow to deploy rust-lang/mdBook site to GitHub Pages.

peaceiris/actions-mdbook: GitHub Actions for mdBook (rust-lang/mdBook)

name: github pages

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  deploy:
    runs-on: ubuntu-22.04
    concurrency:
      group: ${{ github.workflow }}-${{ github.ref }}
    steps:
      - uses: actions/checkout@v4

      - name: Setup mdBook
        uses: peaceiris/actions-mdbook@v2
        with:
          mdbook-version: '0.4.10'
          # mdbook-version: 'latest'

      - run: mdbook build

      - name: Deploy
        uses: peaceiris/actions-gh-pages@v4
        if: ${{ github.ref == 'refs/heads/main' }}
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./book

⭐️ Flutter Web

An exapmle workflow for Flutter web project.

name: github pages

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  deploy:
    runs-on: ubuntu-22.04
    concurrency:
      group: ${{ github.workflow }}-${{ github.ref }}
    steps:
      - uses: actions/checkout@v4

      - name: Setup Flutter
        run: |
          git clone https://github.com/flutter/flutter.git --depth 1 -b beta _flutter
          echo "::add-path::${GITHUB_WORKSPACE}/_flutter/bin"

      - name: Install
        run: |
          flutter config --enable-web
          flutter pub get

      - name: Build
        run: flutter build web

      - name: Deploy
        uses: peaceiris/actions-gh-pages@v4
        if: ${{ github.ref == 'refs/heads/main' }}
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./build/web

⭐️ Elm

An exapmle workflow for Elm.

name: github pages

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  deploy:
    runs-on: ubuntu-22.04
    concurrency:
      group: ${{ github.workflow }}-${{ github.ref }}
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '14'

      - name: Setup Elm
        run: npm install elm --global

      - name: Make
        run: elm make --optimize src/Main.elm

      - name: Move files
        run: |
          mkdir ./public
          mv ./index.html ./public/
        # If you have non-minimal setup with some assets and separate html/js files,
        # provide --output=<output-file> option for `elm make` and remove this step

      - name: Deploy
        uses: peaceiris/actions-gh-pages@v4
        if: ${{ github.ref == 'refs/heads/main' }}
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./public

⭐️ github/personal-website

github/personal-website - Code that'll help you kickstart a personal website that showcases your work as a software developer.

# .github/workflows/github-pages.yml

name: GitHub Pages

on:
  push:
    branches:
      - master
  schedule:
    - cron: '24 */24 * * *'  # Once a day

jobs:
  deploy:
    runs-on: ubuntu-22.04
    concurrency:
      group: ${{ github.workflow }}-${{ github.ref }}
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to GitHub Pages
        uses: peaceiris/actions-gh-pages@v4
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./
          allow_empty_commit: true
          enable_jekyll: true
          cname: github.peaceiris.com

⭐️ Swift Publish

An example workflow for JohnSundell/Publish.

name: GitHub Pages

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  deploy:
    runs-on: ubuntu-22.04
    concurrency:
      group: ${{ github.workflow }}-${{ github.ref }}
    steps:
      - uses: actions/checkout@v4

      - name: Setup JohnSundell/Publish
        run: |
          cd ${HOME}
          git clone --depth=1 https://github.com/JohnSundell/Publish.git
          cd ./Publish
          swift build -c release
          echo "::add-path::${HOME}/Publish/.build/release"

      - run: publish-cli generate

      - name: Deploy to GitHub Pages
        uses: peaceiris/actions-gh-pages@v4
        if: ${{ github.ref == 'refs/heads/main' }}
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./Output

Check more options and examples on GitHub. peaceiris/actions-gh-pages

That's all! Enjoy GitHub Actions and Static Site Generators!

MkDocs Material Boilerplate - Starter Kit

peaceiris — Tue, 26 Feb 2019 09:24:01 +0000

Overview

I released MkDocs Material Boilerplate - Starter Kit 🚀

👉 peaceiris/mkdocs-material-boilerplate - GitHub

MkDocs Material Boilerplate (Starter Kit) - Deploy documentation to platforms (Netlify, GitHub Pages, GitLab Pages, and AWS Amplify Console) with Docker, pipenv, and GitHub Actions.

Issues, pull requests, comments, and GitHub star ⭐️ are welcome❗️

What is MkDocs

MkDocs is one of the best static site generators. We can write our document using markdown and MkDocs builds static site from markdown.

MkDocs Material is one of the MkDocs theme based on Material Design.

Deploy to platforms

We have a lot of choices to manage documentation and deploy it.

GitHub Pages + GitHub +
- mkdocs gh-deploy command
- CircleCI
- GitHub Actions
GitLab Pages + GitLab + GitLab CI/CD
Netlify +
- GitHub
- GitLab
- BitBucket
AWS Amplify Console +
- GitHub
- GitLab
- BitBucket
- AWS CodeCommit

This boilerplate includes build and deploy settings. We can easily publish our documentation right now❗️

👉 Demo site on GitHub Pages (build & deploy with GitHub Actions)

Go to peaceiris/mkdocs-material-boilerplate - GitHub to get more information.

Thanks for reading and enjoy MkDocs❗️

peaceiris / mkdocs-material-boilerplate

MkDocs Material Boilerplate (Starter Kit) - Deploy documentation to hosting platforms (Netlify, GitHub Pages, GitLab Pages, and AWS Amplify Console) with Docker, pipenv, and GitHub Actions.

MkDocs Material
Boilerplate - Starter Kit

Details

Quick start

git clone https://github.com/peaceiris/mkdocs-material-boilerplate.git
cd mkdocs-material-boilerplate
pipenv sync --dev
pipenv shell
inv serve --config-file mkdocs-sample.yml

License

MIT License
The graduate cap icon made by Freepik from www.flaticon.com is licensed by CC 3.0 BY

About Maintainer

peaceiris Homepage

View on GitHub

DEV Community: peaceiris

Start GitHub Actions self-hosted runner with VirtualBox and Vagrant

Overview

Install VirtualBox

Ubuntu 18.04/20.04

macOS

Install Vagrant

Ubuntu 18.04/20.04

macOS

Start VM

Use the VM and Accelerate docker build

Repository

peaceiris / actions-self-hosted-runners

GitHub Actions self-hosted runner on VirtualBox with Vagrant.

actions-self-hosted-runners

Getting Started

Conclusion

The Case Sensitivity of the GitHub Actions Runner

The Problem

The Reason

Can we select case-sensitivity of the GitHub Actions VM?

Suggest Related Links Action

My Workflow

peaceiris / actions-suggest-related-links

A GitHub Action to suggest related or similar issues, documents, and links. Based on the power of NLP and fastText.

Suggest Related Links Action

Table of Contents

Getting Started

Create Workflow

Workflow Overview

Data Collection

Preprocessing

Train Model

Find Similar Issues

Suggest Issues

Submission Category

Additional Resources / Info

Deploy to GitHub Pages with GitHub Actions for Static Site Generator

GitHub Actions for deploying to GitHub Pages

peaceiris / actions-gh-pages

GitHub Actions for GitHub Pages 🚀 Deploy static files and publish your site easily. Static-Site-Generators-friendly.

GitHub Pages Action

Supported Tokens

Getting Started

peaceiris / actions-hugo

GitHub Actions for Hugo ⚡️ Setup Hugo quickly and build your site fast. Hugo extended, Hugo Modules, Linux (Ubuntu), macOS, and Windows are supported.

GitHub Actions for Hugo

Table of Contents

Options

Other Examples

⭐ Static Site Generators with Node.js

⭐️ Static Site Generators with Python

⭐️ mdBook (Rust)

⭐️ Flutter Web

⭐️ Elm

⭐️ github/personal-website

⭐️ Swift Publish

MkDocs Material Boilerplate - Starter Kit

Overview

What is MkDocs

Deploy to platforms

More

peaceiris / mkdocs-material-boilerplate

MkDocs Material Boilerplate (Starter Kit) - Deploy documentation to hosting platforms (Netlify, GitHub Pages, GitLab Pages, and AWS Amplify Console) with Docker, pipenv, and GitHub Actions.

MkDocs MaterialBoilerplate - Starter Kit

Details

Quick start

Links

License

About Maintainer

Use the VM and Accelerate `docker build`

MkDocs Material
Boilerplate - Starter Kit