Extending Buildkite with plugins: HashiCorp Vault

In this post, I'll cover what Buildkite plugins are, how they work in Buildkite and use the Vault secrets plugin as my example. The Vault plugin is the recommended way to integrate with HashiCorp Vault and is a great option for managing and protecting your secrets.

What's a plugin?

Buildkite's CI/CD pipelines support extensibility in so many ways, and plugins play a big role in this composability.

Plugins are small self-contained pieces of functionality that can be included in pipelines so you can customize Buildkite to your specific requirements. You might already be familiar with Jenkins plugins, or CircleCI Orbs, Buildkite plugins are very similar, but are even more flexible.

They can be used to modify the behavior of jobs in your pipeline using one or more of Buildkite's ten job lifecycle hooks. Each hook modifies a different part of the job lifecycle, for example:

• Setting up the environment.
• Checking out the code.
• Running commands.
• Handling artifacts.
• Cleaning up the environment.

Plugins can be open source and available for anyone to use or private and kept in a repository only your organization and agents can access. If you keep them private, they can be in a centralized place or the same repository as your source code.

Using plugins

If you're familiar with Jenkins, you'll be familiar with having a central or web-based plugin management system. Like many other CI/CD tools, Buildkite lets you manage plugins directly in your pipeline definitions. The advantage of this is that they're decentralized, more easily version controlled, and only included in the pipelines that really need them.

You use plugins in pipeline command steps to access a library of commands or to perform actions. To add a plugin to a command step, use the plugins attribute in your pipeline definition. The plugins attribute accepts an array so you can add multiple plugins to the same step.

For example, the following pipeline:

• runs a test.sh script
• if needed, it logs into Docker using the Docker Login plugin
• It then runs the test.sh script inside an app service container using the Docker Compose plugin

steps:
  - command: test.sh
    plugins:
      - docker-login#v1.2.3:
          username: mello
      - docker-compose#v4.14.0: 
          run: app
          image-repository: index.docker.io

Which is the equivalent to running:

docker-compose run app test.sh

Securing your secrets with the Vault secrets plugin

Buildkite is an extremely secure CI/CD tool, you don't store any secrets in Buildkite, we don't have (or want) access to them. Because build agents (aka runners in other CI tools) run on your infrastructure, secrets are only accessed within the boundaries of your environment.

A secret in CI/CD is anything you want to tightly control access to, such as API keys, passwords, and certificates. HashiCorp Vault is an identity-based secrets and encryption management system. Along with securely storing secrets and granting access, it also allows you to create ephemeral/short-lived tokens, and because it's centrally managed it makes revoking and rotating tokens extremely simple should the need arise. You can use Vault in your pipelines to manage your secrets securely.

The Vault secrets plugin is developed and maintained by Buildkite as the recommended way to use Vault in your pipelines. It aims to provide a general solution most teams can use to integrate with Vault.

The plugin allows build agents to authenticate to Vault and acquire pipeline secrets while running a job. It also enables you to create more granular policies for access by agents and pipelines, making it easier to have a better security posture.

The plugin wraps Vault's CLI and allows you to configure multiple authentication methods to acquire a token, including AppRole, AWS, and JWT.

For example, the following pipeline accesses Vault using AppRole authentication:

steps:
  - command: ./run_build.sh
    plugins:
      - vault-secrets#v1.1.0:
          server: "https://my-vault-server"
          path: secret/buildkite
          auth:
            method: "approle"
            role-id: "my-role-id"
            secret-env: "VAULT_SECRET_ID"

Order of operations:

• Authenticate to Vault to get a token that is scoped to the appropriate policies using the configured authentication method.

• Check Vault for any secrets using different paths:

  • env
  • environment
  • private_ssh_key
  • id_rsa_github
  • git-credentials

• Download the secrets and add them to the job environment.

You can use this demo repository to quickly get a local Vault environment running.

Plug'n play

It's best to extract and wrap commonly used or repeated functionality into a plugin, not only will this simplify its inclusion in pipelines, it reduces repetitive code, and makes updating versions or removing and changing plugins across multiple pipelines much simpler.

Many common problems or workflows already have dedicated plugins from Buildkite or the Buildkite community. Check the Plugins directory to look for an existing solution.

If you can't find an existing plugin that does what you need, check out the documentation to write your own. Buildkite plugins are written in Bash and loosely coupled with Buildkite, making them more maintainable and less prone to compatibility issues than in other CI/CD tools.

---