Building a CI pipeline for a Go library with Dagger

Originally published at https://sagikazarmark.hu.

---

I've been playing with Dagger for months now using it in various projects. In this post, I'll share my experience with using Dagger to build CI pipelines for Go libraries.

TL;DR: Check out this repository for a complete example.

Important: I'm going to focus on Go library specific details and will not explain Dagger basic concepts. Please check out the documentation for an introduction into Dagger.

Go library CI

The easiest way to evaluate Dagger for the Go library use case is to compare it to an existing solution. It's probably fair to say that GitHub Actions dominates the CI market for Open Source Software these days, so it makes sense to compare building a pipeline with Dagger to an existing GitHub Actions workflow.

Here is a simple one for a Go library:

name: CI

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest
    strategy:
      matrix:
        go: ['1.16', '1.17', '1.18']

    steps:
      - name: Set up Go
        uses: actions/setup-go@v3
        with:
          go-version: ${{ matrix.go }}

      - name: Checkout code
        uses: actions/checkout@v3

      - name: Test
        run: go test -race -coverprofile=coverage.txt -covermode=atomic ./...

      - name: Upload coverage
        uses: codecov/codecov-action@v3
        if: always()
        with:
          files: coverage.txt

  lint:
    name: Lint
    runs-on: ubuntu-latest

    steps:
      - name: Set up Go
        uses: actions/setup-go@v3
        with:
          go-version: 1.18

      - name: Checkout code
        uses: actions/checkout@v3

      - name: Lint
        uses: golangci/golangci-lint-action@v3

In general terms it consists of the following steps:

• Build matrix to run tests (in parallel) for different environments and settings (eg. multiple Go versions)
• Static analysis and linters
• Publishing analysis results (eg. code coverage) to a third-party service

There are way more complex pipelines out there that run various static analysis tools and integrate with many more services, but most of them fall into one of the three categories above. (Actually, there is a fourth one, but that's hardly ever relevant for a Go library: artifact publishing)

Let's see how we can build a pipeline with Dagger!

Build matrix

Go libraries generally support at least two (or more) Go versions which means CI has to execute tests on all supported versions.

Dagger allows nesting actions, so a naive implementation of a build matrix just repeats the same steps with different Go versions:

test: {
    "1.16": go.#Test & {
        source:  client.filesystem["."].read.contents
        package: "./..."

        _image: go.#Image & {
            version: "1.16"
        }

        input: _image.output
    }

    "1.17": // go.#Test ...
    "1.18": // go.#Test ...
}

The different actions then can be executed either per version (in parallel CI jobs) or at the same time (eg. locally):

dagger do test 1.18 # Run tests for a single Go version
dagger do test # Run tests for all Go versions

Naturally, this isn't the most optimal solution: it doesn't work well with multiple dimensions and it requires a lot of duplication even for a single dimension.

An alternative solution is using templating in CUE. It reduces the amount of code and works well with multiple dimensions:

test: {
    "1.16": _
    "1.17": _
    "1.18": _

    [v=string]: go.#Test & {
        source:  client.filesystem["."].read.contents
        package: "./..."

        _image: go.#Image & {
            version: v
        }

        input: _image.output
    }
}

Admittedly, this is not a real build matrix either as all variations have to be listed manually, but for most of the use cases it's close enough.

Static analysis

The Go ecosystem has an exceptionally large number of static analysis tools and chances are not all of them are going to be available in Dagger Universe (Dagger's central repository of actions).

Writing a custom action allows integrating any arbitrary tool into a Dagger plan.

First, make sure that cue.mod/module.cue contains a module name:

module: "github.com/sagikazarmark/dagger-go-library"

Then create a new CUE file for the tool definition (eg. ci/golangci/lint.cue):

// Lint using golangci-lint
#Lint: {
    // Source code
    source: dagger.#FS

    // golangci-lint version
    version: *"1.46" | string

    // Timeout
    timeout: *"5m" | string

    _image: docker.#Pull & {
        source: "index.docker.io/golangci/golangci-lint:v\(version)"
    }

    go.#Container & {
        name:     "golangci_lint"
        "source": source
        input:    _image.output
        command: {
            name: "golangci-lint"
            flags: {
                run:         true
                "-v":        true
                "--timeout": timeout
            }
        }
    }
}

Generally speaking, an action needs a source code input and some tool specific parameters as inputs.

The action can then be added to the plan:

lint: {
    "go": golangci.#Lint & {
        source:  client.filesystem["."].read.contents
        version: "1.46"
    }
}

Note: GolangCI has been recently added to the alpha channel of Dagger Universe, so you can use that one instead.

Publishing analysis results

It's common for libraries to publish (static) analysis results after test runs to analyze and visualize various code metrics.

One of those metrics is code coverage that tells us how much of the source code was actually executed (covered) by test runs. Code coverage information is generated by the go test tool, so the output of the test step needs to be passed to another step.

Dagger handles these kind of dependencies well, all we need to do is reference the output of one action in another:

test: {
    "1.16": _
    "1.17": _
    "1.18": _

    [v=string]: {
        _test: go.#Test & {
            // ...
            command: flags: {
                "-covermode":    "atomic"
                "-coverprofile": "/coverage.out"
            }

            export: files: "/coverage.out": _
        }
        _coverage: codecov.#Upload & {
            // Merge the coverage file with the source code (for VCS information)
            _write: core.#WriteFile & {
                input:    client.filesystem["."].read.contents
                path:     "/coverage.out"
                contents: _test.export.files."/coverage.out"
            }

            source: _write.output
            file:   "coverage.out"
        }
    }
}

Note: You can find the definition for the Upload action here.

Running it on GitHub Actions

Although we have a plan, it still needs to be executed somewhere. GitHub Actions is a perfect candidate, especially for Open Source Go libraries.

Rewriting the above workflow, we can easily run Dagger actions:

name: Dagger

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest
    strategy:
      matrix:
        go: ['1.16', '1.17', '1.18']

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Dagger
        uses: dagger/dagger-for-github@v3
        with:
          cmds: |
            project update
            do check test go ${{ matrix.go }}

  lint:
    name: Lint
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Dagger
        uses: dagger/dagger-for-github@v3
        with:
          cmds: |
            project update
            do check lint

There is one more change we have to make to our plan: Some actions (like coverage upload) might need detailed VCS information. One option is to pass them manually to each action or let the tools collect them from the environment (eg. environment variables):

client: env: {
    CI:                string | *""
    GITHUB_ACTIONS:    string | *""
    GITHUB_ACTION:     string | *""
    GITHUB_HEAD_REF:   string | *""
    GITHUB_REF:        string | *""
    GITHUB_REPOSITORY: string | *""
    GITHUB_RUN_ID:     string | *""
    GITHUB_SERVER_URL: string | *""
    GITHUB_SHA:        string | *""
    GITHUB_WORKFLOW:   string | *""
}
actions: {
    test: {
        "1.16": _
        "1.17": _
        "1.18": _

        [v=string]: {
            _test: go.#Test & {
                // ...
            }
            _coverage: codecov.#Upload & {
                // ...

                // No need to run coverage upload unless running in CI
                dryRun: client.env.CI != "true"

                env: {
                    GITHUB_ACTIONS:    client.env.GITHUB_ACTIONS
                    GITHUB_ACTION:     client.env.GITHUB_ACTION
                    GITHUB_HEAD_REF:   client.env.GITHUB_HEAD_REF
                    GITHUB_REF:        client.env.GITHUB_REF
                    GITHUB_REPOSITORY: client.env.GITHUB_REPOSITORY
                    GITHUB_RUN_ID:     client.env.GITHUB_RUN_ID
                    GITHUB_SERVER_URL: client.env.GITHUB_SERVER_URL
                    GITHUB_SHA:        client.env.GITHUB_SHA
                    GITHUB_WORKFLOW:   client.env.GITHUB_WORKFLOW
                }
            }
        }
    }
}

Check out the example repository for more details and examples (like caching on GitHub Actions).

Conclusion

The first question that needs to be answered: Is it really worth it?

Well, it depends.

On one hand, building pipelines locally is certainly easier
than doing so on a CI server. It's also comforting to know that if it runs locally, it'll run on the CI server.

On the other hand, CI pipelines for Go libraries tend to be simple and with GitHub Actions the building blocks are as reusable as Dagger actions are. Native CI runs also tend to be a bit faster due to the lack of extra runtime initialization, but that performance hit can be minimized with caching and is less noticeable in large projects.

So it really comes down to personal preference.

My personal favorite feature in Dagger is its portability: I like that I can run and build it on my own machine, but portability really becomes important for complex pipelines and large projects. For a simple Go library, I'll probably stick to Makefiles and GitHub Actions.

Make sure to check out the complete example for more details.