Speeding up Go Modules for Docker and CI

Finally, the Golang world has a built-in, conventional dependency manager in the ecosystem: Go Modules. What began in Go 1.11 as an opt-in feature has become widely adopted by the community, and we are so close to Go 1.13 when Go Modules will be enabled by default. The delightful dilemma of choosing the “best” tool can be finally resolved.

I can’t help but mention two features which are very close to my heart:

– No more $GOPATH imprisonment! In my years of experience, I’d gotten used to storing everything I work on in ~/Projects/ and its subfolders somewhere in the home directory, no matter the programming language. So, being forced to keep my Golang stuff in another specific place and respect SCM url in the path was a real pain, and made routine cd operations feel like such a chore. No longer an issue!
– No more vendoring! Dependency updates don’t produce enormous PR diffs to read, and repositories are lighter. I can just remove the vendor folder from my source code and forget about it.

The migration to Go Modules is pretty simple and won’t take more than a couple of minutes, especially if you use any of the supported package managers to migrate from.

$ go mod init
$ rm vendor/*
$ go test ./...
$ git add .
$ git commit

That’s pretty much it!

With Go Modules your dependencies are not a part of your source code anymore. The toolchain downloads them on its own, keeps modules up-to-date, and caches them locally in $GOPATH/pkg/mod for future use. That sounds perfect for when all of your processes occur in a stateful environment like a laptop, but what about stateless builds in your CI pipeline or Docker? Every now and again Go will download every item in your dependencies and waste your priceless time. Let’s fix that with some caching!

Caching on CI

It’s such a common situation to cache dependencies between builds on CI that some of the services provide a simplified, ecosystem-specific syntax to make it easier. Alas, I haven’t found specific Go Modules caching on popular CIs yet, so let’s do it manually.

If you use TravisCI, it’s very straightforward. Just add those lines to your .travis.yml config and you’re all set:

cache:
  directories:
    - $GOPATH/pkg/mod

Setting up dependency caching on my favorite CircleCI is a little more verbose. Wrap go mod download or your build step in the code below. Golang will take care of the missing dependencies and CircleCI will cache them between builds relying on the content of the go.sum file.

      - restore_cache:
          keys:
            - go-modules-v1-{{ checksum "go.sum" }}
            - go-modules-v1
      # get dependencies here with `go mod download` or implicitly 
      # with `go build` or `go test`
      - save_cache:
          key: go-modules-v1-{{ checksum "go.sum" }}
          paths:
            - "/go/pkg/mod"

Here are the results of boosting of my little project on CircleCI:

Before:

`go test ./...` => 00:20s

After cache warm-up:

Restoring Cache => 00:03s
`go test ./...` => 00:06s
Saving Cache    => 00:00s

Not bad at all: 2x faster CI build, and for free.

Caching in Docker

There are two completely different use cases for how we use Docker: for the development process to isolate an application and its environment, and for packing production builds.

In Development

If you follow Test Driven Development (TDD) caching, Go Modules can significantly increase your development productivity. You definitely know how crucial it is to have as fast a test suite as possible.

Using Docker Compose, cache your modules in a separate volume, and see the performance boost. I saved 20 seconds. Not that bad for a small change!

Here is a minimal docker-compose.yml, simplified for the sake of brevity, with highlighted volumes changes:

version: '3'
services:
  app:
    image: application:0.0.1-development
    build:
      context: .
      dockerfile: docker/development/Dockerfile
    volumes:
      - .:/app
      - go-modules:/go/pkg/mod # Put modules cache into a separate volume
  runner:
    <<: *app
  test:
    <<: *app
    command: go test ./...
volumes:
  go-modules: # Define the volume

In production

For production builds, we can take advantage of the layer caching power. Dependencies change less often than the code itself—let’s make it a separate step in your Dockerfile before the build phase.

# `FROM` and other prerequisites here skipped for the sake of brevity

# Copy `go.mod` for definitions and `go.sum` to invalidate the next layer
# in case of a change in the dependencies
COPY go.mod go.sum ./
# Download dependencies
RUN go mod download

# `RUN go build ...` and further steps

In a nutshell

Introducing Go Modules was an exciting moment and a significant relief to the Golang community. It brought us a lot of excellent features we had been waiting a long time for. Don’t hesitate to try Modules if you haven’t yet. It’s pretty easy to migrate to it, but don’t forget to change your CI or Docker settings to avoid downloading overhead and keep your builds blazing fast.

Comments

[ankush chadha]

Great article. An additional way to speed up the build is how effectively the initial cache can be populated via the GOPROXY protocol that involves downloading go modules via https. There are public GOPROXY available such as GoCenter (gocenter.io) that will further speed up the build process as a lot of git clones are avoided by the client-side.

The added advantage of using GOPROXY is that if there is a need to rebuild the cache, then reproducibility is guaranteed that is not the case if pointing to VCS directly.