Build and release go binaries for Mac and Linux in GitHub Actions using 2 approaches

Ok, let's get started. We choose one of our popular repositories, awsu (Enhanced account switching for AWS, supports Yubikey as MFA source)

kreuzwerker / awsu

Enhanced account switching for AWS, supports Yubikey as MFA source

Amazon Web Services Switch User (awsu)

awsu provides a convenient integration of AWS virtual MFA devices into commandline based workflows. It does use Yubikeys to provide the underlying TOTP one-time passwords but does not rely on additional external infrastructure such as e.g. federation.

There is also a high-level video overview from This Is My Architecture Munich:

[ Installation | Usage | Configuration | Caching | Commands | General multifactor considerations ]

Installation

Production-ready Mac releases can be installed e.g.through brew via kreuzwerker/homebrew-taps:

brew tap kreuzwerker/taps && brew install kreuzwerker/taps/awsu

Linux is only available for download from the release tab. No Windows builds are provided at the moment.

Prequisites

awsu relies on shared credentials files (the same configuration files that other tools such as e.g. the AWS commandline utilities are also using) being configured. The profiles are used to determine

1. which IAM long-term credentials (access key pairs…

View on GitHub

This tool is written in Golang and still used travis-ci as CI. Furthermore, some parts of the release process were still manually, such as uploading the assets to a GitHub release and generating the release notes. We wanted to have this automated.

We started by defining our goals for the hacktoberfest session:

• use the latest version of Golang for building, which is 1.17 when writing this post. This state implied bumping the version of Golang used internally and also removing deprecated package manager, such as godep and/or govendor.
• move from travis to GitHub actions
• provide a suitable config for goreleaser and use it for release. Fortunately, there is already a GitHub action for this tool 🚀
• generate a homebrew formula pointing to a new release and
• update our homebrew-taps automatically 🍺 🤖

My Workflow

The most tricky part was that libraries needed to be installed on the system Linux system. We realized that beforehand the binaries were built on a Mac (with an Intel chip) before, one native and the other in a docker environment with a debian:stretch docker image. As GitHub Actions runs on ubuntu runner, we decided to reproduce the build locally in this environment. This enabled us to go with our first iteration.

Iteration 1 (using docker)

As GitHub Actions allows us to use macos-latest runners, install Docker automatically as it does not exist as for example, for the ubuntu-latest or windows-latest runners. Therefore we created PoC repository docker-gh-action-test to verify this and provide a stable installation for macos.

You can see all the changes in this pull request:

feat(ci): add linux build #57

mavogel posted on Nov 25, 2021

What does this do / why do we need it?

Add the automatic release for the linux_amd64 binary

How this PR fixes the problem?

• builds the binary in the CI
• adds them as prebuilt to goreleaser

What should your reviewer look out for in this PR?

{Please write here}

Check lists

• [x] Test passed
• [x] Coding style (indentation, etc)

Which issue(s) does this PR fix?

fixes #56

View on GitHub

We were happy as it worked as expected. However, our build time was with 11m9s very long. So we explored other options are ended with trying the artifacts feature, which we will talk about
in the second iteration coming now.

Iteration 2 (using matrix builds and artifacts)

We started implementing the solution using 2 stages

1. a build stage with a matrix on ubuntu and macos, where we install the needed dependencies, pre-build the binary and then upload it to the artifacts store.
2. a release stage where we download the artifacts and release them with the prebuilt feature of goreleaser-pro

You can see all the changes in this pull request:

chore: use gh action cache #58

mavogel posted on Nov 26, 2021

improvement of #57

View on GitHub

and a fix an upcoming pull request

fix(ci): path in download artifact #60

mavogel posted on Nov 30, 2021

What does this do / why do we need it?

Fixing #59

How this PR fixes the problem?

Using the correct actions download path. See here

Check lists

• [x] Test passed
• [x] Coding style (indentation, etc)

View on GitHub

This reduced the pipeline runtime from 11m9s down to 1m15s 🎉

Submission Category:

Maintainer Must-Haves

Yaml File or Link to Code

As we described in the previous approach and the logic, here are the concrete implementations of the Actions

Iteration 1 (using docker)

name: goreleaser

on:
  push:
    tags:
      - '*'

jobs:
  binaries:
    runs-on: macos-11
    steps:
      - name: Checkout
        uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - name: Set up Go
        uses: actions/setup-go@v2
        with:
          go-version: 1.17
      - name: Updating and upgrading brew
        run: |
          ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
          brew --version
      - name: Install and start docker
        # https://github.com/docker/for-mac/issues/2359#issuecomment-943131345
        run: |
          brew install --cask docker
          sudo /Applications/Docker.app/Contents/MacOS/Docker --unattended --install-privileged-components
          open -a /Applications/Docker.app --args --unattended --accept-license
          echo "Waiting for docker to be up"
          while ! /Applications/Docker.app/Contents/Resources/bin/docker info &>/dev/null; do echo -n "."; sleep 1; done
      - name: Build linux binary
        run: |
          make build/awsu-linux-amd64
          # as it is the format goreleaser expects
          cp build/awsu-linux-amd64 build/awsu_linux_amd64
      - name: Run GoReleaser
        uses: goreleaser/goreleaser-action@v2
        with:
          distribution: goreleaser-pro
          version: latest
          args: release --rm-dist
        env:
          GORELEASER_KEY: ${{ secrets.GORELEASER_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GORELEASER_TOKEN: ${{ secrets.GORELEASER_TOKEN }}

Iteration 2 (using matrix builds and artifacts)

name: goreleaser

on:
  push:
    tags:
      - '*'

jobs:
  artifact-build:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest]
    steps:
      - name: Checkout
        uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - name: Set up Go
        uses: actions/setup-go@v2
        with:
          go-version: 1.17
      - name: Setup dependencies
        if: matrix.os == 'ubuntu-latest'
        run: |
          sudo apt-get update -q
          sudo apt-get install -qqy build-essential software-properties-common pkg-config wget libpcsclite-dev
      - name: Build darwin
        if: matrix.os == 'macos-latest'
        run: |
          make build/awsu-darwin-amd64
          mv build/awsu-darwin-amd64 build/awsu-macos-latest-amd64
      - name: Build linux
        if: matrix.os == 'ubuntu-latest'
        run: |
          make build/awsu-linux-amd64
          mv build/awsu-linux-amd64 build/awsu-ubuntu-latest-amd64
      - uses: actions/upload-artifact@v2
        with:
          name: awsu-${{ matrix.os }}-amd64
          path: build/awsu-${{ matrix.os }}-amd64

  release-test:
    runs-on: ubuntu-latest
    needs: [artifact-build]
    steps:
      - name: Checkout
        uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - name: Download macos
        uses: actions/download-artifact@v2
        with:
          name: awsu-macos-latest-amd64
          path: build
      - name: Download linux
        uses: actions/download-artifact@v2
        with:
          name: awsu-ubuntu-latest-amd64
          path: build
      - name: Correct goreleaser prebuilt path
        run: |
          # as it is the format goreleaser expects. See .goreleaser.yml -> prebuilt -> path
          mv build/awsu-ubuntu-latest-amd64 build/awsu_linux_amd64
          mv build/awsu-macos-latest-amd64 build/awsu_darwin_amd64
          ls -lash build
      - name: Run GoReleaser
        uses: goreleaser/goreleaser-action@v2
        with:
          distribution: goreleaser-pro
          version: latest
          args: release --rm-dist
        env:
          GORELEASER_KEY: ${{ secrets.GORELEASER_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GORELEASER_TOKEN: ${{ secrets.GORELEASER_TOKEN }}

We hope this will help you folks our there 🎉 Happy to hear more improvements and suggestions or any questions to help clarify the internal logic.

Additional Resources / Info

• The PoC repository for running docker in Github Actions:

  mavogel / docker-gh-action-test

  Running docker in GitHub Actions

  Docker GH action test

  A small test/POC for the docker daemon in GitHub actions.

  See test.yaml

  jobs:
    info:
      runs-on: ${{ matrix.os }}
      strategy:
        fail-fast: false
        matrix:
          os: [ubuntu-latest, macos-latest, windows-latest]
      steps:
        - name: Checkout
          uses: actions/checkout@v2
          with:
            fetch-depth: 0
        - name: Updating and upgrading brew
          if: matrix.os == 'macos-latest'
          run: |
            bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
            brew --version
        - name: Install and start docker
          if: matrix.os == 'macos-latest'
          # https://github.com/docker/roadmap/issues/80#issuecomment-1083441568
          run: |
            # for the docker cli
            brew install docker
            # installed by default
            colima start
        - name: Test docker
          run: |
            docker version
            docker info

  Why

  • I wanted to run docker in windows, macos and linux in GitHub actions to have a minimal POC.
  • 2021-10-28: Currently it's working for docker in version v20.10.8 for macos with the code…

  View on GitHub
• Our GitHub organization kreuzwerker
• The actions used upload-artifact, download-artifact, goreleaser-action
• This post is a successor of our internal hacktoberfest at kreuzwerker