DEV Community: Max

Autoscaling GitLab Runners on Hetzner Cloud

Max — Sun, 06 Dec 2020 19:00:59 +0000

I use a private instance of GitLab a lot and utilize the GitLab CI for most of my private projects. To speed up the CI jobs I wanted to have more GitLab runners available on demand. From our GitLab instance at work I already knew that GitLab supports autoscaling runners with docker-machine. At work we use this in combination with our OpenStack cloud, but for my personal use I was looking for a way to deploy GitLab runners at Hetzner Cloud. Searching the internet for a while led me to this (German) blog post which sounded like a good approach. So I went ahead and started to implement it in a very similar way. Long story short... I am extremly happy with the result! I can get an infinite amount of runners in less then a minute when I need them and they are deleted if I don't need them anymore. This way I only pay when I use them... some would even say that this is pretty much the point of the cloud and on demand resources 😄

The Architecture

GitLab runners can be installed on all kinds of systems and then ask the GitLab server in regular intervals for new jobs they should execute. Docker-machine takes this approach to the next level and the jobs are not executed by the runner directly, instead it creates additional machines on demand and instructs them to run the jobs. If the additional machines idle for a while docker-machine removes them to save resources and money. Originally docker-machine was developed by Docker, but they stopped working on the project. Lucky for us GitLab forked the project and maintains it with minimal patches to ensure that we can still create autoscaling runners with it.

Docker-machine itself does not support Hetzner Cloud out of the box but someone implemented a driver for it. 🙂 We are going to use this Docker image to run docker-machine with Hetzner Cloud support. The image is based on the official runner image and adds the driver to it, which is exactly what we need.

The Runner

How and where we setup the runner is up to us. It doesn't need much resources (remember it does not run the jobs itself) so I just run it on the same server with mit GitLab instance. An easy way to get started is to use docker-compose. Feel free to use any other kind of orchestrator/automation/script to manage it.

A minimal docker-compose.yml file is:

version: '2'
services:
  hetzner-runner:
    image: mawalu/hetzner-gitlab-runner:latest
    volumes:
      - "./hetzner_config:/etc/gitlab-runner"

Let's get this up by running docker-compose up -d. Next thing we can do is to run docker-compose run hetzner-runner register and answer a few interactive prompts, I decided to use Hetzners cpx11 flavor with 2 CPUs and 2GB of memory and the ubuntu-20.04 image. Adjust this depending on the requirements of your jobs. We can now exit our temporary container and should find a config file in ./hetzner_config/config.toml which we can edit to configure the runner further.

concurrent = 50
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Hetzner Autoscale"
  url = "https://git.example.com"
  token = "RUNNER TOKEN FROM GITLAB"
  executor = "docker+machine"
  limit = 10
  [runners.custom_build_dir]
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "OPTIONAL S3 PROVIDER"
      AccessKey = "OPTIONAL S3 CONFIG"
      SecretKey = "OPTIONAL S3 CONFIG"
      BucketName = "OPTIONAL S3 CONFIG"
  [runners.docker]
    tls_verify = false
    image = "ubuntu:20.04"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0
  [runners.machine]
    IdleCount = 0
    IdleTime = 1800
    MachineDriver = "hetzner"
    MachineName = "runner-%s"
    MachineOptions = ["hetzner-api-token=HETZNER API TOKEN", "hetzner-image=ubuntu-20.04", "hetzner-server-type=cpx11"]
  [[runners.machine.autoscaling]]
    Periods = ["* * * * * sat,sun *"]
    IdleTime = 21600

All configuration options are documented in the official docs but let's take a closer look at a few options.

limit = 10: The upper limit of machines created by autoscaling. We should always set this to a reasonable amount for our environment. In my case I don't want to create 100 machines by accident, so I keep it at a low ten. Though, this means if I have more than ten parallel jobs a few have to stay in a pending state for a while until a runner becomes available.
runners.cache: This is completely optional but if you have some S3 compatible object storage available you can configure it here. This speeds CI jobs up because the runners are able to share their cache. You can even use Minio to setup your own S3 compatible storage if you want to. If you don't have S3 storage available, don't worry, just remove this from the config.
/var/run/docker.sock: This is also optional! The Docker socket is usually required when we want to build Docker images with native Docker (docker build ...). The downside of this is that we bypass all security features that we gain by using Docker in the first place. So we can only do this if we trust all people running code in these runners. If you don't need the Docker socket available inside your runners, remove this from the volumes list. An alternative to mounting the socket and using docker build is to use a Docker builder which runs in userspace and does not need the Docker daemon, for example I tinkered around with kaniko in the past. It works but in my experience it takes roughly twice as long as the Docker daemon to build a new image.
runners.machine: That's where it gets interesting. IdleCount = 0 tells docker-machine to not create any spare machines by default. This means if there are no jobs, there won't be any machines (also no costs). Though, if we create new jobs it will take a moment until new machines are created and become available, this leads to a slower pipeline run, but Hetzner Cloud machines boot up really fast — the machines are up and running in less then ten seconds and then afterwards it takes less then a minute for docker-machine to provision the runner on it. For faster results we could increase the IdleCount, e.g. IdleCound = 5 to always have at least five runners available to run new jobs immediately (though we would have to pay five servers permanently). IdleTime = 1800 instructs docker-machine to keep idle runners around for 30 minutes before deleting them. This makes consecutive jobs faster because we don't have to wait for new machines to be provisioned.
runners.machine.autoscaling: We can define this multiple times to override the runners.machine settings for specific time periods. In this example we tell docker-machine to keep idle runners for 6 hours on the weekend. For me this makes sense because I mostly use the runners for personal projects during the weekend. So when I start a few jobs at the weekend I will probably start more, so there is no need to delete idle runners just after 30 minutes. If you want to use this kind of setup in a work environment you probably want other autoscaling settings. For example, a bunch of IdleCount machines and a high IdleTime during the week from morning to evening, less during night and almost none at the weekend.

The costs

What does it cost do run GitLab runners on Hetzner Cloud? Unfortunately there is no easy answer. The most important factor is the machine flavor we choose, the range goes from 0.005€/h (1 CPU, 2GB memory) to 0.095€/h (16 CPUs, 32GB memory). I always find these hourly prices hard to grasp, so let's put it another way: If we run a runner of the smallest instance for a full month we will pay roughly 3€, with the biggest flavor we pay 60€. The cpx11 flavor of our example costs 0.007€/h, which means roughly 4€ per month. In my use cases none of the runners will run for a full month and during most weekdays none will be created whatsoever, so I don't expect more than a few bucks per month. If you set IdleCount = 10 for a full month you will see a difference on the invoice though. One thing to keep in mind while configuring the IdleTime: Hetzner bills the machines hourly, so each started hour costs, no matter if the server runs 1 minute or 59 minutes. So a very small IdleTime probably doesn't make sense because we would create and delete servers more then necessary which means unnecessary costs.

Conclusion

With this fairly easy setup we now have autoscaling on demand GitLab runners on Hetzner Cloud. We only pay what we use and it works super reliable. 🚀

Managing your git repositories with ghq

Max — Sun, 29 Sep 2019 00:14:37 +0000

The more you work with git to manage your code (or other stuff), the messier your directory structure becomes and after a few years you will maybe end up with something like this:

├── code
│   ├── blog
│   └── website
├── repos
│   ├── coolproject
│   └── demo
├── repositories
│   └── website
└── src
    ├── foo
    └── project1337

At least for me this happened. Most of these directories were at least actual git repositories but some were not. Obviously this is a mess and if each of the directories contains a few dozens of repositories it is getting hard to find the right one.

So I was looking for a better way to organize my repositories and a friend told me about ghq. ghq makes it easy to manage all your git repositories and enforces an organized directory structure at the same time. If you work with go you will already know the directory structure as it is exactly the same.

x-motemen / ghq

Remote repository management made easy

ghq(1)

NAME

ghq - Manage remote repository clones

DESCRIPTION

'ghq' provides a way to organize remote repository clones, like go get does. When you clone a remote repository by ghq get, ghq makes a directory under a specific root directory (by default ~/ghq) using the remote repository URL’s host and path.

$ ghq get https://github.com/x-motemen/ghq
# Runs `git clone https://github.com/x-motemen/ghq ~/ghq/github.com/x-motemen/ghq`

You can also list local repositories (ghq list).

SYNOPSIS

ghq get [-u] [-p] [--shallow] [--vcs <vcs>] [--look] [--silent] [--branch] [--no-recursive] <repository URL>|<host>/<user>/<project>|<user>/<project>|<project&gt
ghq list [-p] [-e] [<query>]
ghq create [--vcs <vcs>] <repository URL>|<host>/<user>/<project>|<user>/<project>|<project&gt
ghq root [--all]

COMMANDS

get: Clone a remote repository under ghq root directory (see DIRECTORY STRUCTURES below). If the repository is already cloned to local, nothing will happen unless '-u' ('--update') flag is supplied, in which case the local repository is updated ('git pull --ff-only' eg.) When you use '-p' option, the repository is cloned via…

View on GitHub

Install ghq

Installing ghq is easy. If you use a Linux distribution you can check if ghq is available with your package manger. Otherwise you have a few other options. You can download a current release from GitHub or if you have go installed you can run go get github.com/motemen/ghq. In case you use macOS you can run brew install ghq. To verify your installation just run:

$ ghq root
/home/max/.ghq

Change the ghq root directory

By default ghq organizes all git repositories in ~/.ghq but this can be changed in your ~/.gitconfig file. For example, I do not like the idea of putting all my repositories in a hidden directory like .ghq. Instead I use ~/repos, so the config entry in my ~/.gitconfig file looks like this:

[ghq]
  root = ~/repos
  root = ~/go/src

By defining root=... multiple times ghq uses all directories to look for repositories. This is particularly helpful if you work with go as it organizes repositories in the same way but puts them in ~/go/src by default. If you run ghq list to display all your repositories ghq looks in ~/repos and ~/go/src as well.

Clone a new repository

Instead of running git clone https://github.com/githubtraining/hellogitworld.git to checkout a new repository you now have to run ghq get https://github.com/githubtraining/hellogitworld.git and ghq will use the repository URL to create a proper directory structure:

repos/
└── github.com
    └── githubtraining
        └── hellogitworld
            ├── build.gradle
            ├── fix.txt
            ├── pom.xml
            ├── README.txt
            └ ...

If you clone more repositories from different servers and from different users ghq will create directories to keep everything organized. For example an excerpt of my ~/repos directory looks like this:

├── aur.archlinux.org
│   └── heluxup
│       └── PKGBUILD
├── git.centralserver.de
│   └── max
│       ├── bachelor-thesis
│       ├── dotfiles
│       ├── fotoallerlei
│       ├── k8s
│       ├── telegram-stickers
│       └── wiki
└ github.com
    ├── ekeih
    │   ├── dwdpollen
    │   ├── FrundenBot
    │   ├── hello-github-actions
    │   ├── heluxup
    │   ├── hetzner-deployment
    │   ├── i3-renameworkspace
    │   ├── icinga2telegram
    │   ├── InfiniteWisdomBot
    │   ├── tado
    │   ├── tado-influxdb
    │   ├── taustakuva
    │   └── webhook
    ├── fluxcd
    │   └── helm-operator
    └── grandchild
        └── arch-cleaner

`ghq look` and what's bad about it

Personally, I find this way of organization already a huge benefit of ghq but the major advantage is the way ghq helps to directly jump to the right repository:

$ pwd
/home/max
$ ghq look hello
$ pwd
/home/max/repos/github.com/githubtraining/hellogitworld

This way it is possible to switch between different repositories fast and without the need to know the full paths of them. Unfortunately, this has a downside... the current working directory of a process is part of its environment and a process environment can not be modified by its child processes. If you want to read some more technical background about this I recommend this Stack Overflow answer which explains it in more detail. In the end it means that ghq can not really change the current working directory of your shell... instead it starts a new subshell in the new directory. Technically this works but I find it very annoying that each ghq look ... adds an additional nested shell process end exiting the shell means to return to the previous shell.

answer re: Change parent shell directory from Go

Dec 31 '18

How can I write a Go program that will behave as 'cd ' does?

This is impossible on POSIX systems (even by using any other programming languages).

Because each process, including the parent shell process, has its own current working directory. Hence cd has to be a shell…

Open Full Answer

The developer of ghq suggested to use a shell function to wrap ghq to solve this issue in a GitHub issue. This approach may seem a bit weird but it actually works great and is probably the only solution to really change the current working directory by running cd which is a shell builtin (shell builtins run directly in the shell process and not in a new one, therefore they can modify the environment of the shell process).

Comment for #29

motemen commented on Aug 19, 2014

I'd like to, but I don't think ghq can change the shell's working directory since it is just a command, not a shell function. Any ideas to implement that?

One solution would be providing a shell function like:

ghq () {
    if [ "$1" = look -a -n "$2" ]; then
        cd $(command ghq list -e -p $2)
        return
    fi

    command ghq "$@"
}

View on GitHub

I extended the function a bit and added it to my ~/.bashrc:

ghq () {
  if [ "$1" = look -a -n "$2" ]; then
    local repos=($(command ghq list -p "$2"))
    case ${#repos[@]} in
      0)
        echo 'No repo found.'
        return 1
        ;;
      1)
        cd "${repos[0]}"
        return
        ;;
      *)
        local PS3="Select repo: "
        select reponame in ${repos[@]}; do
          cd "${reponame}"
          return
        done
    esac
  elif [ "$1" = get -a -n "$2" ]; then
    command ghq "$@"
    cd $(command ghq list -e -p "$2")
    return
  fi
  command ghq "$@"
}

Remember to open a new shell after modifying your ~/.bashrc or run source ~/.bashrc to load the changes. So what does it do? Each time you run ghq your shell does not run the real ghq binary (e.g. in /usr/bin/ghq) directly, instead it runs the ghq shell function with the same name.

Line 2-19 wrap the ghq look ... call which means that when you call ghq look foobar the function searches for a repository with foobar in its name and if it finds exactly one it runs cd ... to switch to the directory of it. In case multiple repositories match foobar it offers a selection and then switches to the selected repository.

$ ghq look foobar
No repo found.
$ ghq look fotoallerlei
$ pwd
/home/max/repos/git.centralserver.de/max/fotoallerlei
$ ghq look heluxup
1) /home/max/repos/aur.archlinux.org/heluxup
2) /home/max/repos/github.com/ekeih/heluxup
Select repo: 2
$ pwd
/home/max/repos/github.com/ekeih/heluxup

Line 20-25 wrap the ghq get ... call to switch to the new repository after it is cloned successfully. If ghq is called neither with look nor with get the original ghq binary is called directly.

Wrap-Up

After moving my existing repositories to the directory structure of ghq I started to use it every day. It took me a few days to remember to use ghq get ... instead of git clone ... and ghq look ... instead of cd ~/repos/..., but I like this new workflow very much and can recommend it to everyone who uses multiple git repositories. The shell function to wrap the ghq call to avoid nested shell processes makes it a lot easier to use, so please take a second and add it to your ~/.bashrc if you start to use ghq.

DEV Community: Max

Autoscaling GitLab Runners on Hetzner Cloud

The Architecture

The Runner

The costs

Conclusion

Managing your git repositories with ghq

x-motemen / ghq

Remote repository management made easy

ghq(1)

NAME

DESCRIPTION

SYNOPSIS

COMMANDS

Install ghq

Change the ghq root directory

Clone a new repository

ghq look and what's bad about it

answer re: Change parent shell directory from Go

Comment for #29

Wrap-Up

`ghq look` and what's bad about it