TL;DR 🔍
Explore the step-by-step guide to deploying GitLab on Kubernetes, focusing on the Omnibus package configuration. Learn to set up PostgreSQL, SMTP, Container Registry, Sidekiq, Prometheus metrics, and backups. Discover an alternative with the Glasskube GitLab Kubernetes Operator, simplifying the process and reducing setup time to just 5 minutes.
We Want Your Feedback! 🫶
Share your thoughts in the comments below! Let us know what topics you'd like more content on. If this guide helps, click on the cat and leave a star to support us in creating more developer-centric content. Your feedback matters!
Let´s get started 🏌️
GitLab is an open source DevSecOps platform for software engineering teams.
There are two flavors available for deploying GitLab on your Kubernetes Cluster:
- GitLab Cloud native hybrid
- GitLab package (Omnibus)
GitLab Cloud native hybrid
Deploying the GitLab Cloud native hybrid architecture makes only sense in query specific use-cases. For example - as seen in the GitLab decision tree - if the GitLab instance need to serve at least 3,000 users. For this kind of deployment the GitLab components will be installed separately and in different docker containers. Resulting in minimum requirements of 10 nodes with a total of 19 vCPUs and 60 GB memory, which will result in a couple of thousand dollars cloud costs a month.
So in most cases GitLab Cloud native is not needed and overly complex.
So, how to deploy GitLab Omnibus on Kubernetes?
GitLab Omnibus on Kubernetes
Using the name “omnibus”, GitLab also publishes all-in-one packages, which are available as docker images hub.docker.com/r/gitlab/gitlab-ce, that can be easily configured via environment variables. Doing the configuration right makes it easy to deploy GitLab on Kubernetes.
Following important components should be configured correctly in order to archive a reasonable Kubernetes setup:
- PostgreSQL database
- SMTP configuration
- GitLab container registry on Kubernetes
- Sidekiq, Gitaly & Puma configuration
- Prometheus metrics
- GitLab backups on Kubernetes
GitLab on Kubernetes: Configure an external PostgreSQL database
Creating a PostgreSQL database can be done with the CloudNativePG PostgreSQL Operator. After the installation of the operator a new Postgres cluster can be deployed by applying a Kubernetes CR:
kind: Cluster
apiVersion: postgresql.cnpg.io/v1
metadata:
name: gitlab
spec:
enableSuperuserAccess: false
instances: 2
bootstrap:
initdb:
database: gitlabhq_production
owner: gitlab
storage:
size: 20Gi
It is important that the database is called gitlabhq_production and the configured gitlab database user is the owner of the database. In this case, we created a cluster consisting of two PostgreSQL replicas to ensure that the database remains available, even in case the node running the primary replica fails. This is called high availability (HA).
Now, that we created our very own PostgreSQL cluster, the database included in the omnibus image must be disabled in the gitlab.rb file.
postgresql['enable'] = false
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = 'gitlab-pg-rw'
gitlab_rails['db_password'] = '<your-password>'
Nice, we already have cloud native database for our GitLab installation.
GitLab on Kubernetes: Configure SMTP credentials
In order to make sure your GitLab instance is able to send emails, you need to configure an SMTP server. This also can be done in the gitlab.rb file.
gitlab_rails['smtp_enable'] = ...
gitlab_rails['smtp_address'] = ...
gitlab_rails['smtp_port'] = ...
gitlab_rails['smtp_user_name'] = ...
gitlab_rails['smtp_password'] = ...
gitlab_rails['smtp_tls'] = ...
gitlab_rails['gitlab_email_from'] = ...
A free smtp server for transactional emails can be created for example on the Brevo platform.
GitLab on Kubernetes: Container registry
Among it's many features, GitLab supports serving as a container registry. This is achieved by bundling the reference implementation of the docker container registry, but it is disabled by default as it is quite a hassle to set it up correctly.
Important
It is important to understand that all requests in the GitLab omnibus container will first be processed by an internal nginx server and then distributed to components.
The container registry only works with TLS-encrypted connections, so we need to disable TLS termination by the ingress load balancer and send encrypted traffic directly to the GitLab container. If the NGINX ingress controller is used, this can be done by adding the following annotation to the ingress: nginx.ingress.kubernetes.io/ssl-passthrough: true.
After that, the following gitlab.rb configurations must be applied:
registry['enable'] = true
gitlab_rails['registry_enabled'] = true
registry['token_realm'] = "https://" + ENV['GITLAB_HOST']
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = ENV['GITLAB_REGISTRY_HOST']
gitlab_rails['registry_api_url'] = "http://localhost:5000"
registry_external_url 'https://' + ENV['GITLAB_REGISTRY_HOST']
registry_nginx['redirect_http_to_https'] = true
registry_nginx['listen_port'] = 5443
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/tls.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/tls.key"
registry_nginx['real_ip_trusted_addresses'] = ['10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16']
registry_nginx['server_names_hash_bucket_size'] = 128
Additionally, an S3 bucket (or compatible cloud storage endpoint can be configured so all image layers will not be stored inside a volume, but in a scalable S3
GitLab on Kubernetes: Sidekiq, Gitaly & Puma configuration
Puma (HTTP server for ruby), Sidekiq (job scheduler) and Gitaly (GitLabs Git bridge) can all be started with a custom amount of workers / threads. The best configuration heavily depends on your use-case and the amount of users that need to be supported. A reasonable minimal configuration would be:
puma['worker_processes'] = 2
sidekiq['max_concurrency'] = 9
GitLab on Kubernetes: Prometheus Metrics
The GitLab omnibus package ships with its own prometheus instance inside the image. As a prometheus instance is already present in most Kubernetes clusters the included prometheus can safely be disabled.
prometheus_monitoring['enable'] = false
A ServiceMonitor can be used to tell a running Prometheus instance to monitor the exposed metrics endpoints of GitLabs component on the following ports:
- 8082 (sidekiq)
- 9168 (gitlab)
- 9236 (gitaly)
GitLab on Kubernetes: Backups
Well, backing up GitLab on Kubernetes could be tech guide on its own. The easiest way would be to back up the complete namespace using a backup solution like Velero.
Glasskube GitLab Kubernetes Operator
Meet the Glasskube GitLab Kubernetes Operator - a product of our developer team's frustration with the tedious configuration process, time-consuming setup, and constant troubleshooting associated with GitLab deployment. In response to these challenges, we crafted a solution that simplifies the entire experience. The Glasskube GitLab Kubernetes Operator deploys a fully configured GitLab instance with all features neatly abstracted through a Custom Resource Definition (CRD). Thanks to the Operator, the days of spending excessive time on setup and updates are over. Now, you can effortlessly spin up new GitLab instances within a mere 5 minutes. Try it out and give us feedback!
Install the Glasskube operator
The first step is to install Glasskube via its helm chart:
helm repo add glasskube https://charts.glasskube.eu/
helm repo update
helm install my-glasskube-operator glasskube/glasskube-operator
The full documentation can be found in our Getting started documentation.
Deploy GitLab
Important
A DNS entry must be set on theLoadBalanceror theIngress Host. SSL certificates are
automatically generated byLoadBalancerorcert-managerif aClusterIssueris configured.
gitlab.yaml
apiVersion: "v1"
kind: "Secret"
metadata:
name: "gitlab-smtp"
stringData:
username: "..."
password: "..."
---
apiVersion: v1
kind: Secret
metadata:
name: gitlab-registry-secret
stringData:
accessKey: "..."
secretKey: "..."
---
apiVersion: "glasskube.eu/v1alpha1"
kind: "Gitlab"
metadata:
name: "gitlab"
spec:
host: "gitlab.mycompany.eu"
sshEnabled: true
sshHost: "ssh.gitlab.mycompany.eu"
smtp:
host: "..."
port: 465
fromAddress: "noreply+gitlab@mycompany.eu"
authSecret:
name: "gitlab-smtp"
tlsEnabled: true
registry:
host: "registry.gitlab.mycompany.eu"
storage:
s3:
bucket: gitlab-registry
accessKeySecret:
name: gitlab-registry-secret
key: accessKey
secretKeySecret:
name: gitlab-registry-secret
key: secretKey
region: ...
kubectl apply -f gitlab.yaml
That's it!
The complete custom resource documentation can be found in the Glasskube documentation
about GitLab
Deploy GitLab Runner on Kubernetes
Runners on GitLab are execution agents that power continuous integration and continuous delivery (CI/CD) pipelines.
They're responsible for running jobs, which are the individual steps or tasks within a pipeline.
The Glasskube Gitlab Kubernetes Operator makes it as simple as it gets to add runners to your Gitlab. First a new runner needs to be created via https://{{host}}/admin/runners/new. After that these runner tokens can just be added to gitlab.yaml spec and the Glasskube Kubernetes Operator will automatically spawn and connect these runners with the Gitlab instance.
runners:
- token: glrt-xxxxXX-xxxxxXXXXX # can be generated at https://{{host}}/admin/runners/new
The complete custom resource documentation can be found in the Glasskube documentation about GitLab runner
Now the installation is complete. Updates to the Kubernetes operator will automatically update GitLab.
Conclusion
Deploying a GitLab instance on Kubernetes for less than a thousand users shouldn't be done with GitLabs cloud native hybrid helm charts but rather with a special purposed installation of the GitLab Omnibus package in combination with cloud native infrastructure. The Glasskube Kubernetes operator takes care of that.
Star Glasskube:
Top comments (7)
Omnibus HA on VMs should suffice for 99% of companies out there.
And you do not mention at all the official Helm Chart ? From the documentation and my own experience, it uses moderate resources :
Which is not more than your setup. In my experience, it is easier to install than your tutorial, with basically 2 helm commands :
(You can go as hybrid as you want with it, but I see that your tutorial is not hybrid).
And where is your GitLab Pages server ?
Did I misunderstood your point ? Please enlighten me if so, I've the impression that you reinvented the wheel.
Hi @bcouetil
Thanks for comment, let me clarify some of the points :-)
Our setup easily fits in 4 GB of RAM and 2 vCPUs. (for around 20 Users and 1 runner) without the Kubernetes control plane overhead.
That's a good point that we didn't addressed yet. I just created an issue and we are open to contributions: github.com/glasskube/operator/issu...
Using helm for a quick start makes it super easy to get going, but maintenance will become super hard over time. Especially major PostgreSQL updates, configuring backups and deploying components to multiple (different) namespaces.
To address these issues we started Glasskube as a super easy way to deploy and operator deployments on Kubernetes as we can abstract a lot of that complexity with opinionated best practices.
What additional features do you miss for Glasskube, we are thinking about an installer / GUI already for a long time. And happy to get your input via a discusion / feature request. :-)
Interesting, I will dig into that, thank you for the detailed answer 🙏
Nice article!
Awesome post!
Really cool write up, thanks for sharing!
Good one! Kubernetes can be daunting to start with, so guides are like this are super valuable.