Nucu Labs

Posted on Jan 4

Self Hosting Gitlab

#gitlab #linux #selfhosting

Hello everyone and happy new year!

Introduction

I've been running GitLab as my software forge for the last three weeks and everything was smooth. I replaced my Forgejo
instance with it because it has a better UI and UX. Gitlab is more resource consuming than Forgejo and my 5$/month
VPS turned into 17$/month in order to host and use it without lag.

Well, 204$ a year is a bit too much in my opinion just to host the software forge, so I decided to buy a MiniPC and host it
myself instead, and with the help of Cloudflare I can safely expose it to the internet using tunnels.

The idea behind tunnels is that you run a cloudflare agent on the PC and you won't need to expose ports to the outside.
All traffic is received through the tunnel directly, and since I host my DNS with Cloudflare this solution works well.

My power bill is also low, much lower than my 5$/month initial VPS. I have a smart plug which tracks energy usage, and
I'm using a power cord that powers: Gitlab MiniPC, Old Gaming PC that I use as a gitlab runner, a network switch and overall
I'm averaging about 1KW a day.

This article's focus is a how to host Gitlab and perform basic operations for maintenance.

Gitlab

Gitlab is a platform similar to Github, I call it software forge, the term is taken from Forgejo.
Other than managing Git repositories Gitlab comes with a handful of features like:

container registry
gitlab pages (static website hosting)
mattermost (a chat software like Slack)
and of course many more...

Linux Package

You can host Gitlab using various ways such as Kubernetes or Docker Containers. I prefer to host it using the
Linux package method. My distribution of choice is Alma Linux.

Installation

I won't cover this here since Gitlab has amazing docs, and you may have a different distro:

Install the Linux package on AlmaLinux and RHEL-compatible distributions

Note that Gitlab has a free version and a paid version. You need to install the community edition.

Once Gitlab is installed take note of the root account's password located at /etc/gitlab/initial_root_password.

You may verify that everything is running by executing gitlab-ctl status.

Automatic Updates

I use automatic updates on my system. You can enable them on AlmaLinux using:

sudo dnf install dnf-automatic
sudo systemctl enable --now dnf-automatic.timer

Configuration

I have some opinionated configs regarding the Gitlab instance, the configuration file that I will provide next does
the following things:

I disable metrics because I don't monitor my Gitlab's server resource usage.
I do not use SSL certificates and Let's Encrypt because I want Cloudflare do provide SSL for me.
Nginx will only listen on port 80.
I use Gitlab bundled Nginx. That means that I do not install the nginx package on the system.
I customize the nginx to serve my static blog.
I use Scaleway's transactional email service for email.
I use Gitlab pages and the container registry.

# ➜  ~ head -n100 /etc/gitlab/gitlab.rb
external_url 'https://gitlab.nuculabs.dev'
registry_external_url 'https://registry.nuculabs.dev'

letsencrypt['enable'] = false
gitlab_rails['time_zone'] = 'Europe/Bucharest'
nginx['client_max_body_size'] = '2g'
nginx['custom_nginx_config'] = "include /etc/gitlab/nginx/sites-enabled/*.conf;"
nginx['listen_https'] = false
nginx['listen_port'] = 80 
nginx['redirect_http_to_https'] = false
registry_nginx['listen_https'] = false
registry_nginx['listen_port'] = 80
registry_nginx['redirect_http_to_https'] = false
gitlab_rails['gitlab_ssh_host'] = 'git.nuculabs.dev'

# Gitlab Mattermost
# mattermost_external_url 'https://mattermost.nuculabs.dev'
mattermost['enable'] = false
# mattermost_nginx['redirect_http_to_https'] = false
# mattermost_nginx['listen_port'] = 80
# mattermost_nginx['listen_https'] = false

# Gitlab Pages
pages_external_url 'https://pages.nuculabs.dev'
pages_nginx['listen_https'] = false
pages_nginx['listen_port'] = 80
gitlab_pages['namespace_in_path'] = true

# Disable Prometheus and its exporters
prometheus['enable'] = false
prometheus_monitoring['enable'] = false
sidekiq['metrics_enabled'] = false
alertmanager['enable'] = false
node_exporter['enable'] = false
redis_exporter['enable'] = false
postgres_exporter['enable'] = false
gitlab_exporter['enable'] = false
puma['exporter_enabled'] = false
gitlab_kas['enable'] = false


# Email config
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.tem.scaleway.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "xxx"
gitlab_rails['smtp_password'] = "xxx"
gitlab_rails['gitlab_email_from'] = 'no-reply@example.com'
gitlab_rails['gitlab_email_reply_to'] = 'no-reply@example.com'
gitlab_rails['smtp_domain'] = "xx"
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['smtp_enable_starttls_auto'] = true

You can adapt my gitlab.rb and override yours at /etc/gitlab/gitlab.rb.

After you've edited the gitlab.rb config you need to tell Gitlab to reconfigure itself by running:
sudo gitlab-ctl reconfigure.

Also, don't forget to create the custom nginx config folder.

mkdir -p /etc/gitlab/nginx/sites-enabled/

Backups

I back up the instance daily. You can back up the following directories:

/etc/gitlab
/var/opt/gitlab/backups
/etc/gitlab/nginx/sites-enabled/

For backups I use a script with a systemd timer and rclone. Rclone is a very flexible and configurable
tool, and it can be configured with a lot of services. I use it mainly with web dav, smb and Google Drive.

You may use my backup script, it is placed in /root/backup.sh.

Note: that for the script to work you will need to configure rclone and replace nas:/home/gitlab/gitlab with your specific path.

You can configure rclone interactively by typing rclone config.

#!/bin/bash

gitlab-backup create

rclone sync /etc/gitlab nas:/home/gitlab/gitlab --progress --log-file=rclone-nas.log
rclone sync /var/opt/gitlab/backups nas:/home/gitlab/gitlab-backups --progress --log-file=rclone-nas.log
rclone sync /etc/gitlab/nginx/sites-enabled/ nas:/home/gitlab/gitlab-nginx --progress --log-file=rclone-nas.log

ls -t /var/opt/gitlab/backups | tail -n +4 | xargs -I {} rm -f /var/opt/gitlab/backups/{}

Then the systemd services need to be placed in the /etc/systemd/system directory.

# gitlab-backup.timer
[Unit]
Description=Gitlab Backup Timer

[Timer]
# Defines when the service should be activated
# OnCalendar= Specifies a calendar event.
# 'daily' is an alias for '*-*-* 00:00:00'. We use 02:00:00 (2 AM).
OnCalendar=daily
# OR: OnCalendar=*-*-* 02:00:00

# Defines a random delay after the specified time to prevent
# simultaneous resource spikes if you have many timers.
# We'll use a 30-minute window.
RandomizedDelaySec=30m

# Ensures the timer will execute shortly after the system boots
# up if it missed a scheduled run while the system was off.
Persistent=true

[Install]
WantedBy=timers.target

# gitlab-backup.service
[Unit]
Description=Gitlab backup service
RequiresMountsFor=/root

[Service]
User=root
Group=root

# The type of process. 'oneshot' is for a script that runs once and exits.
Type=oneshot

# The command to execute (the full path to your script)
ExecStart=/bin/bash /root/backup.sh

If you've created the files in the current directory run the following commands to copy them to the right directory
and to enable the timer.

cp gitlab-backup.* /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable gitlab-backup.timer
sudo systemctl start gitlab-backup.timer

Restoring from backup

To restore from a backup you need to ensure that the gitlab-ce package version matches the version used when creating the backup, otherwise it won't work.

Essentially I configure rclone and do a sync from the backup destination to the local system:

rclone sync nas:/home/gitlab/gitlab /etc/gitlab --progress --log-file=rclone-nas.log
rclone sync nas:/home/gitlab/gitlab-backups /var/opt/gitlab/backups --progress --log-file=rclone-nas.log
rclone sync nas:/home/gitlab/gitlab-nginx /etc/gitlab/nginx/sites-enabled/ --progress --log-file=rclone-nas.log

Then I stop the Gitlab services:

gitlab-ctl stop puma
gitlab-ctl stop sidekiq

And then I restore my backup. You need to omit _gitlab_backup.tar from the BACKUP env.

➜  ~ ls /var/opt/gitlab/backups 
1767219541_2026_01_01_18.7.0_gitlab_backup.tar  1767304984_2026_01_02_18.7.0_gitlab_backup.tar  1767391444_2026_01_03_18.7.0_gitlab_backup.tar

gitlab-backup restore BACKUP=1767219541_2026_01_01_18.7.0

In a matter of minutes you should have a restored and working gitlab instance. After the restore command finishes you
will need to start the services again:

gitlab-ctl start puma
gitlab-ctl start sidekiq

Nginx

In order to host my blog I use Gitlab's bundled nginx. If you're using my gitlab.rb file you can create the custom
nginx directory if it doesn't exist and place the configuration files in there.

mkdir -p /etc/gitlab/nginx/sites-enabled/

Here are the contents of blog.conf which hosts my blog:

# cat /etc/gitlab/nginx/sites-enabled/blog.conf
server {
    server_name nuculabs.dev;
    listen 80;

    root /blog;
    index index.html; # Hugo generates HTML

    location / {
        try_files $uri $uri/ =404;
    }
}

server {
    server_name www.nuculabs.dev blog.nuculabs.dev;
    listen 80;

    return 301 https://nuculabs.dev$request_uri;
}

As you can see I'm only listening on port 80. SSL is handled by Cloudflare so I don't have to spend time managing certificates
or troubleshooting Let's Encrypt.

Here's another example for a simple redirect:

server {
    server_name forge.nuculabs.dev;
    listen 80;
    return 301 https://gitlab.nuculabs.dev$request_uri;
}

You can hup nginx in order to pick-up new files: gitlab-ctl hup nginx.

Troubleshooting SSH

If everything is configured correctly you should have no issues cloning and pushing repositories through SSH.

Sometimes the directory's security context gets messed up, and you may have to restore it by running:

sudo semanage fcontext -a -t ssh_home_t "/var/opt/gitlab/.ssh(/.*)?"
sudo restorecon -Rv /var/opt/gitlab/.ssh

Note that Cloudflare will only work through HTTP/HTTPS and SSH is not supported. You will be able to access
Gitlab through SSH if you host it on the local network or if you set up a VPN.

Cloudflared

To create Cloudflare Tunnel we need
to install cloudflared.

In your Cloudflare dashboard navigate to Zero Trust -> Networks -> Manage Tunnels -> Add a tunnel.

You can give the tunnel a name and you will get installation instructions for your OS. For AlmaLinux I got:

# Add cloudflared.repo to /etc/yum.repos.d/ 
curl -fsSl https://pkg.cloudflare.com/cloudflared.repo | sudo tee /etc/yum.repos.d/cloudflared.repo

#update repo
sudo yum update

# install cloudflared
sudo yum install cloudflared

Next you need to start the cloudflare service and install the systemd files, you can do it by running:

sudo cloudflared service install <token>

The <token> will be displayed in the installation instructions.

After that's done all you need is to create a route and Cloudflare will update the DNS entries automatically and you can
access your Gitlab instance via HTTPS.