Deploying gitlab on Docker Swarm

Update: You may want to see the instructions on the gitlab documentation.

Docker Swarm is a container orchestration system that is very easy to setup and to get started with, it is built in the Docker engine and can be set up in a few minutes. But deploying a service like Gitlab can be a bit tricky.

In this tutorial I will show how to deploy Gitlab on Docker Swarm while trying to mimic a production environment.

But Kubernetes won the container orchestration war!

I have heard that since Docker announced support for Kubernetes and that Swarm will be discontinued, but I believe the two solutions will coexist (at least I hope so). Kubernetes is an excellent solution but it is not very easy to get started with, and setting up a new cluster can be quite challenging, specially if you have just a few servers available (remember that you need an etcd cluster too).

Kubernetes is still an excellent choice and my guess is that Swarm might become a façade for Kubernetes.

Before we begin...

There are a few things that we will need before we can get to Gitlab:

• A proxy to forward the requests to the correct services
• Postgres and Redis
• Prometheus to collect metrics from Gitlab and Grafana to visualize these metrics
• An NFS4 share to store any persistent data we need

The Proxy

For the proxy, we can use Traefik. We just need to create a new network on our Swarm cluster a start the Traefik service:

$ docker network create --driver overlay proxy
$ docker service create \
    --name traefik \
    --constraint=node.role==manager \
    --publish 80:80 --publish 8080:8080 \
    --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
    --network proxy \
    traefik \
    --docker \
    --docker.swarmmode \
    --docker.domain=localtest.me \
    --docker.watch \
    --web

We first created the proxy network so any service attached to this network will be reachable by Traefik. By mounting the Docker socket into the Traefik service, we allow it to watch for when new services are deployed and to reconfigure itself. The argument --docker.domain=[localtest.me](http://readme.localtest.me/) will instruct Traefik to proxy {service_name}.localtest.me to the corresponding service (and you can override the default settings).

Something to be noted is that the domain "localtest.me" is a service that redirects to 127.0.0.1. Pretty handy since do not have to mess up with /etc/hosts.

Setting up NFS

First we need to install the NFS server:

# On Debian/Ubuntu
apt-get install -y nfs-kernel-server

# On RedHat/CentOS
yum install -y nfs-utils

# Start the NFS server and enable it on startup
systemctl enable --now nfs-server

For the NFS share the will need a directory structure like the following:

/srv/gitlab-swarm/
├── gitlab
│   ├── config
│   ├── data
│   └── logs
├── grafana
├── postgres
└── prometheus

We can create it using the following shell:

mkdir -p /srv/gitlab-swarm && \
mkdir -p /srv/gitlab-swarm/gitlab/{data,logs,config} && \
mkdir -p /srv/gitlab-swarm/postgres && \
mkdir -p /srv/gitlab-swarm/grafana && \
mkdir -p /srv/gitlab-swarm/prometheus && \
chmod -R 777 /srv/gitlab-swarm

Then we need to create the directory /exports/gitlab-swarm and mount srv/gitlab-swarm onto it (this is required for NFS version 4):

mkdir -p /exports/gitlab-swarm
mount --bind /srv/gitlab-swarm /exports/gitlab-swarm

Setup the NFS share by editing /etc/exports:

# /etc/exports

/exports/               *(rw,sync,fsid=0,crossmnt,no_subtree_check)
/exports/gitlab-swarm   *(rw,sync,no_root_squash,no_subtree_check)

Now we reload the NFS configuration:

exportfs -ra

update 2018-08-18
As pointed out by Alexey Petushkov, in order to make the nfs mounts persist after reboots, we need to add the following line to /etc/fstab:

/srv/gitlab-swarm/      /exports/gitlab-swarm/  none    bind

Remember that we need the NFS client installed on each node of the cluster:

# On Debian/Ubuntu
apt-get install -y nfs-common

# On RedHat/CentOS
yum install -y nfs-utils

To test if the NFS configuration is correct, we can try mounting the share:

mkdir /var/tmp/test-nfs && \
mount -t nfs4 127.0.0.1:/gitlab-swarm /var/tmp/test-nfs && \
grep nfs4 /proc/mounts | cut -d ' ' -f 1,2,3 && \
umount /var/tmp/test-nfs

Running the commands above should output this:

127.0.0.1:/gitlab-swarm /var/tmp/test-nfs nfs4

Building the stack

Configuration files

To deploy our Gitlab stack first we need to create the configuration files for the services we are deploying.

The first one is for Gitlab itself (gitlab.rb). The Docker image version we are using (10.3.3) is based on the omnibus installation and contains all the services needed for gitlab, including postgres, redis and prometheus. We will disable these services and set gitlab to look for them in other containers:

# gitlab.rb

external_url 'http://gitlab.local'
registry_external_url 'http://registry.gitlab.local'

# Disable services
postgresql['enable'] = false
redis['enable'] = false
prometheus['enable'] = false
postgres_exporter['enable'] = false
redis_exporter['enable'] = false

# Postgres settings
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_encoding'] = "unicode"

# database service will be named "postgres" in the stack
gitlab_rails['db_host'] = "postgres" 
gitlab_rails['db_database'] = "gitlab"
gitlab_rails['db_username'] = "gitlab"
gitlab_rails['db_password'] = "gitlab"

# Redis settings
# redis service will be named "redis" in the stack
gitlab_rails['redis_host'] = "redis"

# Prometheus exporters
node_exporter['listen_address'] = '0.0.0.0:9100'
gitlab_monitor['listen_address'] = '0.0.0.0'
gitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
gitlab_workhorse['prometheus_listen_addr'] = "0.0.0.0:9229"

The Prometheus configuration file to setup metrics collection from gitlab:

# prometheus.yaml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # gitlab monitor
  - job_name: 'gitlab_monitor'
    static_configs:
      - targets: ['gitlab:9168']

  # gitlab sidekiq
  - job_name: 'gitlab_sidekiq'
    metrics_path: /sidekiq
    static_configs:
      - targets: ['gitlab:9168']

  # gitlab process
  - job_name: 'gitlab_process'
    metrics_path: /process
    static_configs:
      - targets: ['gitlab:9168']

  # gitlab pages
  - job_name: 'gitlab_pages'
    static_configs:
      - targets: ['gitlab:9235']

  # gitaly
  - job_name: gitaly
    static_configs:
      - targets: ['gitlab:9236']

  # gitlab workhorse
  - job_name: workhorse
    static_configs:
      - targets: ['gitlab:9229']

The Grafana configuration file:

[database]
path = "/data/grafana.db"

[session]
provider = "redis"
provider_config = "addr=redis:6379,prefix=grafana:"

Services

Now we can start defining the services of our stack. Let's begin with Gitlab itself:

# stack.yaml

version: "3.4"

services:
  gitlab:
    image: "gitlab/gitlab-ce:10.3.3-ce.0"
    volumes:
      - "gitlab_data:/var/opt/gitlab"
      - "gitlab_logs:/var/log/gitlab"
      - "gitlab_config:/etc/gitlab"
    ports:
      - "2222:22"
    configs:
      - source: "gitlab.rb"
        target: "/etc/gitlab/gitlab.rb"
    networks:
      - default
      - proxy
    deploy:
      labels:
        traefik.port: "80"
        traefik.frontend.rule: "Host:gitlab.localtest.me"
        traefik.docker.network: "proxy"

volumes:
  gitlab_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/gitlab/data"
  gitlab_logs:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/gitlab/logs"
  gitlab_config:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/gitlab/config"

configs:
  gitlab.rb:
    file: "./gitlab.rb"

networks:
  proxy:
    external: true

The most notable thing here is the volumes. Docker supports mounting NFS volumes using the "local" driver. The volumes are mounting to 127.0.0.1 because the NFS server is running on the same machine as the my 1 node cluster.

Besides that, we are adding some labels on gitlab service to tell Traefik to use the hostname "gitlab.localtest.me" and port 80 when proxying to this service, and we also needed to tell traefik wich network to use to reach the gitlab service. We needed to set the two networks, "proxy" to be reached with traefik and "default" to copmmunicate with the other services on this stack. We also used a swarm config, which is like a swarm secret, just not encrypted (you can use a secret if you want).

Since we disabled postgres, redis and prometheus in gitlab.rb, we need to setup those services too:

# stack.yaml

services:
  gitlab:
    # ...
  redis:
    image: "redis:4.0.6-alpine"
  postgres:
    image: "postgres:10.1-alpine"
    volumes:
      - "postgres_data:/data"
    environment:
      POSTGRES_USER: gitlab
      POSTGRES_PASSWORD: gitlab
      PGDATA: /data
      POSTGRES_DB: gitlab
  prometheus:
    image: "prom/prometheus:v2.0.0"
    command: "--config.file=/prometheus.yaml --storage.tsdb.path /data"
    volumes:
      - "prometheus_data:/data"
    configs:
      - prometheus.yaml
    networks:
      - default
      - proxy
    deploy:
      labels:
        traefik.port: 9090
        traefik.frontend.rule: "Host:prometheus.localtest.me"
        traefik.docker.network: "proxy"

volumes:
  # ...
  postgres_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: :/gitlab-swarm/postgres
  prometheus_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/prometheus"

configs:
  gitlab.rb: # ...
  prometheus.yaml:
    file: "./prometheus.yaml"

Similarly to the gitlab service, we created NFS volumes to the Postgres and Prometheus services and added the Traefik labels and config file to Prometheus service.

Now to the Grafana service:

# stack.yaml

services:
  # ...
  grafana:
    image: "grafana/grafana:4.6.3"
    environment:
      GF_PATHS_CONFIG: "/grafana.ini"
    configs:
      - grafana.ini
    volumes:
      - "grafana_data:/data"
    networks:
      - default
      - proxy
    deploy:
      labels:
        traefik.port: 3000
        traefik.frontend.rule: "Host:grafana.localtest.me"
        traefik.docker.network: "proxy"

volumes:
  # ...
  grafana_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/grafana"

configs:
  # ...
  grafana.ini:
    file: "./grafana.ini"

Putting it all together we will have stack like this:

# stack.yaml

version: "3.4"

services:
  gitlab:
    image: "gitlab/gitlab-ce:10.3.3-ce.0"
    volumes:
      - "gitlab_data:/var/opt/gitlab"
      - "gitlab_logs:/var/log/gitlab"
      - "gitlab_config:/etc/gitlab"
    ports:
      - "2222:22"
    configs:
      - source: "gitlab.rb"
        target: "/etc/gitlab/gitlab.rb"
    networks:
      - default
      - proxy
    deploy:
      labels:
        traefik.port: 80
        traefik.frontend.rule: "Host:gitlab.localtest.me"
        traefik.docker.network: "proxy"
  redis:
    image: "redis:4.0.6-alpine"
  postgres:
    image: "postgres:10.1-alpine"
    volumes:
      - "postgres_data:/data"
    environment:
      POSTGRES_USER: "gitlab"
      POSTGRES_PASSWORD: "gitlab"
      PGDATA: "/data"
      POSTGRES_DB: "gitlab"
  prometheus:
    image: "prom/prometheus:v2.0.0"
    command: "--config.file=/prometheus.yaml --storage.tsdb.path /data"
    volumes:
      - "prometheus_data:/data"
    configs:
      - prometheus.yaml
    networks:
      - default
      - proxy
    deploy:
      labels:
        traefik.port: 9090
        traefik.frontend.rule: "Host:prometheus.localtest.me"
        traefik.docker.network: "proxy"
  grafana:
    image: grafana/grafana:4.6.3
    environment:
      GF_PATHS_CONFIG: "/grafana.ini"
    configs:
      - grafana.ini
    volumes:
      - "grafana_data:/data"
    networks:
      - default
      - proxy
    deploy:
      labels:
        traefik.port: 3000
        traefik.frontend.rule: "Host:grafana.localtest.me"
        traefik.docker.network: "proxy"

volumes:
  gitlab_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/gitlab/data"
  gitlab_logs:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/gitlab/logs"
  gitlab_config:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/gitlab/config"
  postgres_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/postgres"
  prometheus_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/prometheus"
  grafana_data:
    driver: local
    driver_opts:
      type: nfs4
      o: "addr=127.0.0.1"
      device: ":/gitlab-swarm/grafana"

configs:
  gitlab.rb:
    file: "./gitlab.rb"
  prometheus.yaml:
    file: "./prometheus.yaml"
  grafana.ini:
    file: "./grafana.ini"

networks:
  proxy:
    external: true

Deploying the stack

To deploy our stack, we use the docker command line

docker stack deploy -c stack.yaml gitlab

It may take some time, but when all the services are ready, you can access localhost:8080 to see the Traefik dashboard:

And if you access gitlab.localtest.me you get to our Gitlab instance running on Docker Swarm!

Going to grafana.localtest.me, you can login using "admin" for both username and password, click in "Add datasource" and create a new datasource of type "Prometheus" and with the url as "http://prometheus:9090":

Then you got to "Menu (grafana icon) > Dashboards > Import" and import the Gitlab-Monitor dashboard using the datasource we just created and we will have some graphics of gitlab activity:

Comments

[danielfrancora]

Hi,

Sorry for the newbie question but, how do I change the volumes you created using nfs to a local driver path?
I'm using GlusterFS as a fileshare and I'm having some trouble to put your docker-compose to work because the volume mount type.

I've tryed:
grafana_data:
driver: local
driver_opts:
device: "/opt/local/docker/docker-data/gitlab/grafana"
type: none
o: bind

But I get error on mount.

Can you help?

Thx in advance

[Livio Ribeiro]

Does the directory "/opt/local/docker/docker-data/gitlab/grafana" already exist? In my tests, I could only make it work when the directory to be mounted already exists on the host.

[danielfrancora]

Yes, the directory exists. I created all the directory structure you pointed in the beginning.

[Alexey Petushkov]

Hi Livio,
Thank you for your article. One thing to add:

Add line to /etc/fstab

/srv/gitlab-swarm/      /exports/gitlab-swarm/  none    bind

In other case, it will fail to start after reboot (due to NFS shares)

[Livio Ribeiro]

Thanks! I've updated the post with your suggestion.

[azaars]

Hi Livio,

This is an excellent article indeed. Thanks for sharing.

I'm going to try it out and so, I should be able to scale it up by running

docker service scale mystack_gitlab=3

Will it actually work in a cluster this way?

[Livio Ribeiro]

I never tried that, so I cannot say if it would work.

You can find out about high availability on Gitlab's documentation. The stack I showed is not far from it.

[Nick Parlow]

Hi Livio,
Thanks for this excellent article. really easy to follow, and i've now got gitlab up and running on a three node swarm... :D

any ideas on how to introduce more gitlab front end containers? i've tried following the article here:
docs.gitlab.com/ce/administration/...

but the second node just bounces up and down.

the gitlab article implies that the second and subsequent servers should only need access to the shared secrets, and otherwise use the same confg; they already have acess t othose because the gitlab data directory is on the shared nfs path....

any ideas?

[Livio Ribeiro]

You need to get the logs from the container to see what went wrong.

I just tested this gitlab deploy (although using gitlab 11) and it worked with 3 replicas.

[Nick Parlow]

right - still couldn't get subsequent containers to be stable, even with gitlab 11... but when i rearranged my shared storage, made tw oseparate gitlab services and altered the stack.yaml file, it works:
/srv/gitlab-swarm/
├── gitlab1
│   ├── config
│   ├── data
│   └── logs
├── gitlab2
│   ├── config
│   ├── data
│   └── logs
├── gitlabshared
│   └── data
│   ├── gitdata
│   ├── ssh
│   ├── gitlabrails
│   │ ├── uploads
│   │ └── shared
│   └── gitlabci
│   └── builds
├── grafana
├── postgres
└── prometheus

volumes:
gitlab1_data:
drive: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlab/data"
gitlab1_logs:
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlab/logs"
gitlab1_config:
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlab/config"
gitlab2_data:
drive: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlab/data"
gitlab2_logs:
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlab/logs"
gitlab2_config:
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlab/config"
gitlab_gitdata
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlabshared/data"
gitlab_ssh
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlabshared/ssh"
gitlab_gitrailsupload
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlabshared/data/gitlabrails/upload"
gitlab_gitrailsshared
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlabshared/gitlabrails/shared"
gitlab_gitcibuilds
driver: local
driver_opts:
type: nfs4
o: "addr=10.17.17.10"
device: ":/gitlab-swarm/gitlabshared/gitlabci/builds"

services:
gitlab1:
image: "gitlab/gitlab-ce:10.3.3-ce.0"
volumes:
- "gitlab1_data:/var/opt/gitlab"
- "gitlab1_logs:/var/log/gitlab"
- "gitlab1_config:/etc/gitlab"
- "gitlab_gitdata:/var/opt/gitlab/gitdata"
- "gitlab_ssh:/var/opt/gitlab/.ssh"
- "gitlab_gitrailsupload:/var/opt/gitlab/gitlab-rails/uploads"
- "gitlab_gitrailsshared:/var/opt/gitlab/gitlab-rails/shared"
- "gitlab_gitcibuilds:/var/opt/gitlab/gitlab-ci/builds"

gitlab2:
image: "gitlab/gitlab-ce:10.3.3-ce.0"
volumes:
- "gitlab2_data:/var/opt/gitlab"
- "gitlab2_logs:/var/log/gitlab"
- "gitlab2_config:/etc/gitlab"
- "gitlab_gitdata:/var/opt/gitlab/gitdata"
- "gitlab_ssh:/var/opt/gitlab/.ssh"
- "gitlab_gitrailsupload:/var/opt/gitlab/gitlab-rails/uploads"
- "gitlab_gitrailsshared:/var/opt/gitlab/gitlab-rails/shared"
- "gitlab_gitcibuilds:/var/opt/gitlab/gitlab-ci/builds"

also need to add the shared secrets stuff into gitlab.rb for both services

gitlab_shell['secret_token'] = 'fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860'
gitlab_rails['otp_key_base'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa'
gitlab_rails['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d'
gitlab_rails['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964'

(these are the ones from the gitlab docs, obviously they're not the ones i used. i span up a single node stack, copied the data from the secrets.json file, and then put those into my gitlab.rb file.

thanks for the brilliant article - it helped me a lot. now to make postgres HA. :|

[Ijaz ahmad]

Hi ,

thanks for sharing,

I have mounted an nfs share from a production nfs filer/server on a VM , there is a specific user that has RW access to that share , how can I use that share in setting up this stack , where in this compose configuration I should specific the user credentials so that docker machines can use that.

And how can I make this work if I want to land each service on a different docker host , and what changes should be made on NFS related. NFS share is mounted on each machine with the same user credentials.

thanks
ijaz

[Livio Ribeiro]

Sorry but I cannot help, I don't know much about NFS and never had the chance to use this kind of setup in production.

The "o" option in the volume driver configuration can receive any nfs configuration (as far as I know), maybe you can do something with this and the server setup.

[Rob]

Does this also run a private docker registry in the gitlab container?

[Livio Ribeiro]

Since the storage is set up, then I believe yes