Philip Miglinci for Glasskube

Posted on Nov 29, 2023 • Edited on Jan 30 • Originally published at glasskube.eu

🚀 GITLAB ON KUBERNETES: The Ultimate Deployment Guide! 🌟

#kubernetes #opensource #devops #gitlab

TL;DR 🔍

Explore the step-by-step guide to deploying GitLab on Kubernetes, focusing on the Omnibus package configuration. Learn to set up PostgreSQL, SMTP, Container Registry, Sidekiq, Prometheus metrics, and backups. Discover an alternative with the Glasskube GitLab Kubernetes Operator, simplifying the process and reducing setup time to just 5 minutes.

We Want Your Feedback! 🫶

Share your thoughts in the comments below! Let us know what topics you'd like more content on. If this guide helps, click on the cat and leave a star to support us in creating more developer-centric content. Your feedback matters!

Let´s get started 🏌️

GitLab is an open source DevSecOps platform for software engineering teams.

There are two flavors available for deploying GitLab on your Kubernetes Cluster:

GitLab Cloud native hybrid
GitLab package (Omnibus)

GitLab Cloud native hybrid

Deploying the GitLab Cloud native hybrid architecture makes only sense in query specific use-cases. For example - as seen in the GitLab decision tree - if the GitLab instance need to serve at least 3,000 users. For this kind of deployment the GitLab components will be installed separately and in different docker containers. Resulting in minimum requirements of 10 nodes with a total of 19 vCPUs and 60 GB memory, which will result in a couple of thousand dollars cloud costs a month.

So in most cases GitLab Cloud native is not needed and overly complex.

So, how to deploy GitLab Omnibus on Kubernetes?

GitLab Omnibus on Kubernetes

Using the name “omnibus”, GitLab also publishes all-in-one packages, which are available as docker images hub.docker.com/r/gitlab/gitlab-ce, that can be easily configured via environment variables. Doing the configuration right makes it easy to deploy GitLab on Kubernetes.

Following important components should be configured correctly in order to archive a reasonable Kubernetes setup:

PostgreSQL database
SMTP configuration
GitLab container registry on Kubernetes
Sidekiq, Gitaly & Puma configuration
Prometheus metrics
GitLab backups on Kubernetes

GitLab on Kubernetes: Configure an external PostgreSQL database

Creating a PostgreSQL database can be done with the CloudNativePG PostgreSQL Operator. After the installation of the operator a new Postgres cluster can be deployed by applying a Kubernetes CR:

kind: Cluster
apiVersion: postgresql.cnpg.io/v1
metadata:
  name: gitlab
spec:
  enableSuperuserAccess: false
  instances: 2
  bootstrap:
    initdb:
      database: gitlabhq_production
      owner: gitlab
  storage:
    size: 20Gi

It is important that the database is called gitlabhq_production and the configured gitlab database user is the owner of the database. In this case, we created a cluster consisting of two PostgreSQL replicas to ensure that the database remains available, even in case the node running the primary replica fails. This is called high availability (HA).
Now, that we created our very own PostgreSQL cluster, the database included in the omnibus image must be disabled in the gitlab.rb file.

postgresql['enable'] = false

gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = 'gitlab-pg-rw'
gitlab_rails['db_password'] = '<your-password>'

Nice, we already have cloud native database for our GitLab installation.

GitLab on Kubernetes: Configure SMTP credentials

In order to make sure your GitLab instance is able to send emails, you need to configure an SMTP server. This also can be done in the gitlab.rb file.

gitlab_rails['smtp_enable'] = ...
gitlab_rails['smtp_address'] = ...
gitlab_rails['smtp_port'] = ...
gitlab_rails['smtp_user_name'] = ...
gitlab_rails['smtp_password'] = ...
gitlab_rails['smtp_tls'] = ...
gitlab_rails['gitlab_email_from'] = ...

A free smtp server for transactional emails can be created for example on the Brevo platform.

GitLab on Kubernetes: Container registry

Among it's many features, GitLab supports serving as a container registry. This is achieved by bundling the reference implementation of the docker container registry, but it is disabled by default as it is quite a hassle to set it up correctly.

Important
It is important to understand that all requests in the GitLab omnibus container will first be processed by an internal nginx server and then distributed to components.

The container registry only works with TLS-encrypted connections, so we need to disable TLS termination by the ingress load balancer and send encrypted traffic directly to the GitLab container. If the NGINX ingress controller is used, this can be done by adding the following annotation to the ingress: nginx.ingress.kubernetes.io/ssl-passthrough: true.

After that, the following gitlab.rb configurations must be applied:

registry['enable'] = true
gitlab_rails['registry_enabled'] = true
registry['token_realm'] = "https://" + ENV['GITLAB_HOST']
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = ENV['GITLAB_REGISTRY_HOST']
gitlab_rails['registry_api_url'] = "http://localhost:5000"

registry_external_url 'https://' + ENV['GITLAB_REGISTRY_HOST']
registry_nginx['redirect_http_to_https'] = true
registry_nginx['listen_port'] = 5443
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/tls.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/tls.key"
registry_nginx['real_ip_trusted_addresses'] = ['10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16']
registry_nginx['server_names_hash_bucket_size'] = 128

Additionally, an S3 bucket (or compatible cloud storage endpoint can be configured so all image layers will not be stored inside a volume, but in a scalable S3

GitLab on Kubernetes: Sidekiq, Gitaly & Puma configuration

Puma (HTTP server for ruby), Sidekiq (job scheduler) and Gitaly (GitLabs Git bridge) can all be started with a custom amount of workers / threads. The best configuration heavily depends on your use-case and the amount of users that need to be supported. A reasonable minimal configuration would be:

puma['worker_processes'] = 2
sidekiq['max_concurrency'] = 9

GitLab on Kubernetes: Prometheus Metrics

The GitLab omnibus package ships with its own prometheus instance inside the image. As a prometheus instance is already present in most Kubernetes clusters the included prometheus can safely be disabled.

prometheus_monitoring['enable'] = false

A ServiceMonitor can be used to tell a running Prometheus instance to monitor the exposed metrics endpoints of GitLabs component on the following ports:

8082 (sidekiq)
9168 (gitlab)
9236 (gitaly)

GitLab on Kubernetes: Backups

Well, backing up GitLab on Kubernetes could be tech guide on its own. The easiest way would be to back up the complete namespace using a backup solution like Velero.

Glasskube GitLab Kubernetes Operator

Meet the Glasskube GitLab Kubernetes Operator - a product of our developer team's frustration with the tedious configuration process, time-consuming setup, and constant troubleshooting associated with GitLab deployment. In response to these challenges, we crafted a solution that simplifies the entire experience. The Glasskube GitLab Kubernetes Operator deploys a fully configured GitLab instance with all features neatly abstracted through a Custom Resource Definition (CRD). Thanks to the Operator, the days of spending excessive time on setup and updates are over. Now, you can effortlessly spin up new GitLab instances within a mere 5 minutes. Try it out and give us feedback!

Install the Glasskube operator

The first step is to install Glasskube via its helm chart:

helm repo add glasskube https://charts.glasskube.eu/
helm repo update
helm install my-glasskube-operator glasskube/glasskube-operator

The full documentation can be found in our Getting started documentation.

Deploy GitLab

Important
A DNS entry must be set on the LoadBalancer or the Ingress Host. SSL certificates are
automatically generated by LoadBalancer or cert-manager if a ClusterIssuer is configured.

gitlab.yaml

apiVersion: "v1"
kind: "Secret"
metadata:
  name: "gitlab-smtp"
stringData:
  username: "..."
  password: "..."
---
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-registry-secret
stringData:
  accessKey: "..."
  secretKey: "..."
---
apiVersion: "glasskube.eu/v1alpha1"
kind: "Gitlab"
metadata:
  name: "gitlab"
spec:
  host: "gitlab.mycompany.eu"
  sshEnabled: true
  sshHost: "ssh.gitlab.mycompany.eu"
  smtp:
    host: "..."
    port: 465
    fromAddress: "noreply+gitlab@mycompany.eu"
    authSecret:
      name: "gitlab-smtp"
    tlsEnabled: true
  registry:
    host: "registry.gitlab.mycompany.eu"
    storage:
      s3:
        bucket: gitlab-registry
        accessKeySecret:
          name: gitlab-registry-secret
          key: accessKey
        secretKeySecret:
          name: gitlab-registry-secret
          key: secretKey
        region: ...

kubectl apply -f gitlab.yaml

That's it!
The complete custom resource documentation can be found in the Glasskube documentation
about GitLab

Deploy GitLab Runner on Kubernetes

Runners on GitLab are execution agents that power continuous integration and continuous delivery (CI/CD) pipelines.
They're responsible for running jobs, which are the individual steps or tasks within a pipeline.

The Glasskube Gitlab Kubernetes Operator makes it as simple as it gets to add runners to your Gitlab. First a new runner needs to be created via https://{{host}}/admin/runners/new. After that these runner tokens can just be added to gitlab.yaml spec and the Glasskube Kubernetes Operator will automatically spawn and connect these runners with the Gitlab instance.

  runners:
    - token: glrt-xxxxXX-xxxxxXXXXX # can be generated at https://{{host}}/admin/runners/new

The complete custom resource documentation can be found in the Glasskube documentation about GitLab runner

Now the installation is complete. Updates to the Kubernetes operator will automatically update GitLab.

Conclusion

Deploying a GitLab instance on Kubernetes for less than a thousand users shouldn't be done with GitLabs cloud native hybrid helm charts but rather with a special purposed installation of the GitLab Omnibus package in combination with cloud native infrastructure. The Glasskube Kubernetes operator takes care of that.

Star Glasskube:

`glasskube/glasskube`

Top comments (7)

Benoit COUETIL 💫 • Nov 29 '23

Omnibus HA on VMs should suffice for 99% of companies out there.

And you do not mention at all the official Helm Chart ? From the documentation and my own experience, it uses moderate resources :

A cluster with a total of at least eight virtual CPUs and 30GB of RAM is recommended.

Which is not more than your setup. In my experience, it is easier to install than your tutorial, with basically 2 helm commands :

helm repo add gitlab https://charts.gitlab.io/
helm install gitlab gitlab/gitlab \
  --set global.hosts.domain=DOMAIN \
  --set certmanager-issuer.email=me@example.com

(You can go as hybrid as you want with it, but I see that your tutorial is not hybrid).

And where is your GitLab Pages server ?

Did I misunderstood your point ? Please enlighten me if so, I've the impression that you reinvented the wheel.

Philip Miglinci • Nov 30 '23

Hi @bcouetil

Thanks for comment, let me clarify some of the points :-)

A cluster with a total of at least eight virtual CPUs and 30GB of RAM is recommended.
Which is not more than your setup.

Our setup easily fits in 4 GB of RAM and 2 vCPUs. (for around 20 Users and 1 runner) without the Kubernetes control plane overhead.

And where is your GitLab Pages server ?

That's a good point that we didn't addressed yet. I just created an issue and we are open to contributions: github.com/glasskube/operator/issu...

Did I misunderstood your point ? Please enlighten me if so, I've the impression that you reinvented the wheel.

Using helm for a quick start makes it super easy to get going, but maintenance will become super hard over time. Especially major PostgreSQL updates, configuring backups and deploying components to multiple (different) namespaces.

To address these issues we started Glasskube as a super easy way to deploy and operator deployments on Kubernetes as we can abstract a lot of that complexity with opinionated best practices.

What additional features do you miss for Glasskube, we are thinking about an installer / GUI already for a long time. And happy to get your input via a discusion / feature request. :-)