Ephemeral self-hosted GitHub actions runners

Alex — Thu, 15 Aug 2024 10:35:10 +0000

Many people who use GitHub not only as a code repository but also as a CI system have encountered the need to run CI operations on their own machines. This article covers how to run actions runner in docker without saving state and with rootless-docker.

To run actions-runner in Docker, we'll use the following solution: https://github.com/myoung34/docker-github-actions-runner. It's basically a bash wrapper over the binary from GitHub, which works quite well. The first thing to look at is the security section of the project's readme:

### Security

It is known that environment variables are not safe from
 exfiltration. If you are using this runner make sure that any
 workflow changes are gated by a verification process (in the actions
 settings) so that malicious PR's cannot exfiltrate these.

And it's really true, just run a simple action using this self-hosted runner to see for yourself (might not work, depends on installation):

name: self-hosted-runner-default
on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]

jobs:
  basic-info:
    runs-on: self-hosted
    steps:
      - name: test env vars
        run: |
          sudo xargs -0 -L1 -a /proc/1/environ | grep APP_ID && echo "APP_ID found" || echo "APP_ID not found"

As a result we can see the APP_ID (I removed the ID itself from the screenshot):

As we can see, it is easy for an attacker to obtain an access token or application credentials.
For convenience, the wrapper generates the token in the container itself, so sensitive data remains in the container. However, it is possible to solve this problem by moving the token retrieval scripts out of the container.

The next important step is the ability to run docker-in-docker. By default, the repo offers to throw a docker socket inside the container or run the container in the privileged mode. Running docker as root without the userns-remap option essentially gives the process root privileges out of the box. For example, you can run the container inside of the action with the -v /:/host option and access the host filesystem. If you do enable userns-remap, getting docker.sock to work will be quite a challenge.

To solve this problem, we can run another docker container in parallel, which will run docker-daemon and provide its socket to the github-runner container. Unfortunately, simply passing a socket through a shared volume is not enough, and we are helped by the actions-runner-controller, or rather a comment in one of its PRs: https://github.com/actions/actions-runner-controller/pull/2919#issuecomment-1737868125.

This comment shows you what settings you need to make before you can run the runner and the non-privileged dind-container on it.
There are two ways to run the runner:

Create a github-runner user on the host and add it to the docker group. Then run the dind-rootless container in --privileged mode. Since we are running the container as a rootless user, the --privileged flag will do no harm.
Create a github-runner user on the host, run docker-daemon from it (https://docs.docker.com/engine/security/rootless/#install) and then run dind-rootless.

The second option seems preferable to me, as even the parent process is run by the user, not root. All that remains is to automate all this, wrap it in a systemd unit and run it.

Scripts can be found in my github repo: https://github.com/alvicsam/github-runner-docker-ephemeral

Few assumptions:

Scripts work only for runners that are registered with a github app
The default storage driver for dind is vfs, which is slow if you have an image with many layers. There is a comment to enable fuse-overlayfs which makes dind faster. Unfortunately, I couldn't get it to work with the overlay2 storage driver (as far as I know, it requires permission to use mount(), which the github-runner user doesn't have).

DEV Community: Alex

Ephemeral self-hosted GitHub actions runners