Homebrew and Private GitHub Repositories

I recently developed an internal CLI tool for my organization to use and, of course, wanted to make it easy for the other devs to install. That means I wanted to make it installable from Homebrew, since most, if not all, of my colleagues use it to manage their installed applications. I was able to tap a private repo by setting the HOMEBREW_GITHUB_API_TOKEN environment variable with a GitHub access token, but I was getting 404 errors from curl when installing. The documentation and information I could find by searching was either non-existent or outdated, so I figured I would get what worked for me out there for others to use.

Background

The tool I wrote is a little CLI tool written in go. When I tag a commit on main with a version semver, our CI tool uses GoReleaser to build binaries for various architectures, create a GitHub release, and update the Homebrew formula. GoReleaser is definitely not necessary, but makes things incredibly easy. Here's my .goreleaser.yaml for reference:

before:
  hooks:
    # You may remove this if you don't use go modules.
    - go mod tidy
builds:
  - env:
      - CGO_ENABLED=0
    goos:
      - linux
      - windows
      - darwin
    goarch:
      - amd64
      - arm64
    ignore:
      - goos: windows
        goarch: arm64
archives:
  -
    replacements:
      amd64: x86_64
      darwin: Darwin
      linux: Linux
    format_overrides:
      - goos: windows
        format: zip
brews:
  -
    tap:
      owner: myorg
      name: myrepo
    download_strategy: GitHubPrivateRepositoryReleaseDownloadStrategy
    custom_require: "lib/custom_download_strategy"
    commit_author:
      name: My Name
      email: [e-mail]
    folder: HomebrewFormula
checksum:
  name_template: 'checksums.txt'
snapshot:
  name_template: "{{ incpatch .Version }}-next"
changelog:
  sort: asc
  filters:
    exclude:
      - '^docs:'
      - '^test:'

I did not create a separate repo for the Homebrew formula and instead just put the formula in the tool's repo in a directory named HomebrewFormula. This makes the tap command a bit longer, but I'm fine with that so I don't have to have an extra repo for each tool I want to deploy with Homebrew.

GoReleaser creates gzips for each platform/arch (zip for Windows) named like ${toolname}_${semver_without_leading_v}_${platform}_${arch}.tar.gz. This is important for our Homebrew download strategy.

The Homebrew Sauce

For whatever reason, you can't curl a release asset from a private repo even with a valid access token. So we need some code to use the GitHub API to get the asset's API URL. This comes in the form of a custom download strategy.

I found some old code that did just what's needed but it used some Homebrew functions that have moved or changed. So after some more digging, trial, and error I was able to get it working with the current version of Homebrew (3.3.12 as of writing this).

HomebrewFormula/lib/custom_download_strategy.rb

require "download_strategy"

# S3DownloadStrategy downloads tarballs from AWS S3.
# To use it, add `:using => :s3` to the URL section of your
# formula.  This download strategy uses AWS access tokens (in the
# environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`)
# to sign the request.  This strategy is good in a corporate setting,
# because it lets you use a private S3 bucket as a repo for internal
# distribution.  (It will work for public buckets as well.)
class S3DownloadStrategy < CurlDownloadStrategy
  def initialize(url, name, version, **meta)
    super
  end

  def _fetch(url:, resolved_url:, timeout:)
    if url !~ %r{^https?://([^.].*)\.s3\.amazonaws\.com/(.+)$} &&
       url !~ %r{^s3://([^.].*?)/(.+)$}
      raise "Bad S3 URL: " + url
    end

    bucket = Regexp.last_match(1)
    key = Regexp.last_match(2)

    ENV["AWS_ACCESS_KEY_ID"] = ENV["HOMEBREW_AWS_ACCESS_KEY_ID"]
    ENV["AWS_SECRET_ACCESS_KEY"] = ENV["HOMEBREW_AWS_SECRET_ACCESS_KEY"]

    begin
      signer = Aws::S3::Presigner.new
      s3url = signer.presigned_url :get_object, bucket: bucket, key: key
    rescue Aws::Sigv4::Errors::MissingCredentialsError
      ohai "AWS credentials missing, trying public URL instead."
      s3url = url
    end

    curl_download s3url, to: temporary_path
  end
end

# GitHubPrivateRepositoryDownloadStrategy downloads contents from GitHub
# Private Repository. To use it, add
# `:using => :github_private_repo` to the URL section of
# your formula. This download strategy uses GitHub access tokens (in the
# environment variables `HOMEBREW_GITHUB_API_TOKEN`) to sign the request.  This
# strategy is suitable for corporate use just like S3DownloadStrategy, because
# it lets you use a private GitHub repository for internal distribution.  It
# works with public one, but in that case simply use CurlDownloadStrategy.
class GitHubPrivateRepositoryDownloadStrategy < CurlDownloadStrategy
  require "utils/formatter"
  require "utils/github"

  def initialize(url, name, version, **meta)
    super
    parse_url_pattern
    set_github_token
  end

  def parse_url_pattern
    unless match = url.match(%r{https://github.com/([^/]+)/([^/]+)/(\S+)})
      raise CurlDownloadStrategyError, "Invalid url pattern for GitHub Repository."
    end

    _, @owner, @repo, @filepath = *match
  end

  def download_url
    "https://#{@github_token}@github.com/#{@owner}/#{@repo}/#{@filepath}"
  end

  private

  def _fetch(url:, resolved_url:, timeout:)
    curl_download download_url, to: temporary_path
  end

  def set_github_token
    @github_token = ENV["HOMEBREW_GITHUB_API_TOKEN"]
    unless @github_token
      raise CurlDownloadStrategyError, "Environmental variable HOMEBREW_GITHUB_API_TOKEN is required."
    end

    validate_github_repository_access!
  end

  def validate_github_repository_access!
    # Test access to the repository
    GitHub.repository(@owner, @repo)
  rescue GitHub::HTTPNotFoundError
    # We only handle HTTPNotFoundError here,
    # becase AuthenticationFailedError is handled within util/github.
    message = <<~EOS
      HOMEBREW_GITHUB_API_TOKEN can not access the repository: #{@owner}/#{@repo}
      This token may not have permission to access the repository or the url of formula may be incorrect.
    EOS
    raise CurlDownloadStrategyError, message
  end
end

# GitHubPrivateRepositoryReleaseDownloadStrategy downloads tarballs from GitHub
# Release assets. To use it, add `:using => :github_private_release` to the URL section
# of your formula. This download strategy uses GitHub access tokens (in the
# environment variables HOMEBREW_GITHUB_API_TOKEN) to sign the request.
class GitHubPrivateRepositoryReleaseDownloadStrategy < GitHubPrivateRepositoryDownloadStrategy
  def initialize(url, name, version, **meta)
    super
  end

  def parse_url_pattern
    url_pattern = %r{https://github.com/([^/]+)/([^/]+)/releases/download/([^/]+)/(\S+)}
    unless @url =~ url_pattern
      raise CurlDownloadStrategyError, "Invalid url pattern for GitHub Release."
    end

    _, @owner, @repo, @tag, @filename = *@url.match(url_pattern)
  end

  def download_url
    "https://api.github.com/repos/#{@owner}/#{@repo}/releases/assets/#{asset_id}"
  end

  private

  def _fetch(url:, resolved_url:, timeout:)
    # HTTP request header `Accept: application/octet-stream` is required.
    # Without this, the GitHub API will respond with metadata, not binary.
    curl_download download_url, "--header", "Accept: application/octet-stream", "--header", "Authorization: token #{@github_token}", to: temporary_path
  end

  def asset_id
    @asset_id ||= resolve_asset_id
  end

  def resolve_asset_id
    release_metadata = fetch_release_metadata
    assets = release_metadata["assets"].select { |a| a["name"] == @filename }
    raise CurlDownloadStrategyError, "Asset file not found." if assets.empty?

    assets.first["id"]
  end

  def fetch_release_metadata
    release_url = "https://api.github.com/repos/#{@owner}/#{@repo}/releases/tags/#{@tag}"
    GitHub::API.open_rest(release_url)
  end
end

# ScpDownloadStrategy downloads files using ssh via scp. To use it, add
# `:using => :scp` to the URL section of your formula or
# provide a URL starting with scp://. This strategy uses ssh credentials for
# authentication. If a public/private keypair is configured, it will not
# prompt for a password.
#
# @example
#   class Abc < Formula
#     url "scp://example.com/src/abc.1.0.tar.gz"
#     ...
class ScpDownloadStrategy < AbstractFileDownloadStrategy
  def initialize(url, name, version, **meta)
    super
    parse_url_pattern
  end

  def parse_url_pattern
    url_pattern = %r{scp://([^@]+@)?([^@:/]+)(:\d+)?/(\S+)}
    if @url !~ url_pattern
      raise ScpDownloadStrategyError, "Invalid URL for scp: #{@url}"
    end

    _, @user, @host, @port, @path = *@url.match(url_pattern)
  end

  def fetch
    ohai "Downloading #{@url}"

    if cached_location.exist?
      puts "Already downloaded: #{cached_location}"
    else
      system_command! "scp", args: [scp_source, temporary_path.to_s]
      ignore_interrupts { temporary_path.rename(cached_location) }
    end
  end

  def clear_cache
    super
    rm_rf(temporary_path)
  end

  private

  def scp_source
    path_prefix = "/" unless @path.start_with?("~")
    port_arg = "-P #{@port[1..-1]} " if @port
    "#{port_arg}#{@user}#{@host}:#{path_prefix}#{@path}"
  end
end

class DownloadStrategyDetector
  class << self
    module Compat
      def detect(url, using = nil)
        strategy = super
        require_aws_sdk if strategy == S3DownloadStrategy
        strategy
      end

      def detect_from_url(url)
        case url
        when %r{^s3://}
          S3DownloadStrategy
        when %r{^scp://}
          ScpDownloadStrategy
        else
          super(url)
        end
      end

      def detect_from_symbol(symbol)
        case symbol
        when :github_private_repo
          GitHubPrivateRepositoryDownloadStrategy
        when :github_private_release
          GitHubPrivateRepositoryReleaseDownloadStrategy
        when :s3
          S3DownloadStrategy
        when :scp
          ScpDownloadStrategy
        else
          super(symbol)
        end
      end
    end

    prepend Compat
  end
end

There are some additional download strategies that I have not tested but left in the file just in case I needed them in the future. Then in our Homebrew formula we can just reference this file and tell Homebrew to use our GitHubPrivateRepositoryReleaseDownloadStrategy.

HomebrewFormula/mytool.rb

# typed: false
# frozen_string_literal: true

# This file was generated by GoReleaser. DO NOT EDIT.
require_relative "lib/custom_download_strategy"
class Mytool < Formula
  desc ""
  homepage ""
  version "1.1.5"

  on_macos do
    if Hardware::CPU.arm?
      url "https://github.com/myorg/mytool/releases/download/v1.1.5/mytool_1.1.5_Darwin_arm64.tar.gz", :using => GitHubPrivateRepositoryReleaseDownloadStrategy
      sha256 "abc123..."

      def install
        bin.install "mytool"
      end
    end
    if Hardware::CPU.intel?
      url "https://github.com/myorg/mytool/releases/download/v1.1.5/mytool_1.1.5_Darwin_x86_64.tar.gz", :using => GitHubPrivateRepositoryReleaseDownloadStrategy
      sha256 "qwerty987..."

      def install
        bin.install "mytool"
      end
    end
  end

  on_linux do
    if Hardware::CPU.arm? && Hardware::CPU.is_64_bit?
      url "https://github.com/myorg/mytool/releases/download/v1.1.5/mytool_1.1.5_Linux_arm64.tar.gz", :using => GitHubPrivateRepositoryReleaseDownloadStrategy
      sha256 "f00bar..."

      def install
        bin.install "mytool"
      end
    end
    if Hardware::CPU.intel?
      url "https://github.com/myorg/mytool/releases/download/v1.1.5/mytool_1.1.5_Linux_x86_64.tar.gz", :using => GitHubPrivateRepositoryReleaseDownloadStrategy
      sha256 "xyz543..."

      def install
        bin.install "mytool"
      end
    end
  end
end

Then all that's needed is to run the following commands to install:

HOMEBREW_GITHUB_API_TOKEN=ghp_abc123...
brew tap myorg/mytool https://github.com/myorg/mytool
brew install mytool