Setting up a "confidential" GitLab🦊🔒

#tutorial #devops #opensource #tooling

Hey there. If you ever programmed something "sensitive" or, like me, are managing private meeting notes with Git, you probably know this uneasy feeling when pushing to GitHub. It's simply impossible to tell who may get access to your private repos. There could be malicious insiders leaking stuff or there could be bugs allowing other users to access data etc.

This uneasy feeling is one of the primary reasons for people to use self-managed GitLab. Especially companies concerned about leakage of IP like to run GitLab on-prem. However, maintaining on-prem infrastructure is expensive, tedious, and often not even an option.

Confidential computing to the rescue!

Recent server CPUs from AMD and Intel come with so called confidential computing features. With these, sensitive workloads can be kept encrypted during runtime in memory and are shielded from the rest of a system. Not even system administrators, cloud provider employees or privileged attackers can access workloads protected this way. And this property can even be verified remotely 🤯

Confidential computing effectively lets you create your own runtime-encrypted and isolated "on-prem" on the public cloud. It's like having your cake and eating it 🍰

If you want to learn more about confidential computing, I recently published a whitepaper on it. The Confidential Computing Consortium also has some good resources.

How to make GitLab🦊 "confidential"?

Ok, now that we sort of know what confidential computing is - how do we apply it (correctly) to GitLab and the software it relies on, e.g., PostgreSQL, NGINX, and Redis?

Easy! I recently posted about our open-source project Constellation. Constellation is the first confidential Kubernetes distribution. Think Rancher Kubernetes Engine (RKE) or RedHat OpenShift for confidential computing.

Constellation encrypts and shields entire K8s clusters with the help of confidential computing. Basically, Constellation ensures that all data in a cluster is always encrypted (at runtime, at rest, and in transit) and that the cluster cannot be accessed from the infrastructure.

So making GitLab "confidential" is as simple as running it with Constellation on confidential computing enabled hardware, which is available in Azure and GCP.

Running GitLab🦊 on Constellation💫

The first step is to download and install the Constellation CLI.

Next, a Constellation cluster can be created in a few steps. In essence:

constellation config generate azure
constellation create --control-plane-nodes 1 --worker-nodes 2 -y
constellation init
export KUBECONFIG="$PWD/constellation-admin.conf"

We can now connect to the cluster with kubectl or other tools using the auto-generated constellation-admin.conf. The config ensures that the connection "confidential" and terminates inside the correct cluster. (No man-in-the-middle possible.)

Installing GitLab (manually)

We want our "confidential" GitLab to be accessible from the Internet under its own URL. For this, we need to install and configure external-dns in our freshly created cluster.

Further, we need to install Constellation's CSI driver for Azure. The driver ensures that all data written to cloud storage from the cluster gets encrypted automatically. With the upcoming v2.3 release of Constellation, it will be installed by default.

Finally, we can install GitLab via the official Helm chart, which also installs Redis, PostgreSQL, NGINX, and other components in the cluster.

Installing GitLab (automated)

We also created a custom Helm chart that installs and configures all of the above. This can be used as follows:

export GODADDY_API_KEY=<your creds here>
export GODADDY_SECRET_KEY=<your creds here>
export TARGET_DOMAIN=<your domain, e.g. gitlab.edgeless.systems>
export TLS_ISSUER_EMAIL=<your e-mail address>
helm dependency update ./gitlab
helm upgrade gitlab-infra ./gitlab --install --namespace default --set infra.enabled=true --set apiKey=$GODADDY_API_KEY --set secretKey=$GODADDY_SECRET_KEY
helm upgrade gitlab-app ./gitlab --install --timeout 600s --set app.enabled=true --set gitlab.global.hosts.domain=$TARGET_DOMAIN --set gitlab.certmanager-issuer.email=$TLS_ISSUER_EMAIL --namespace gitlab --create-namespace

Note that the above assumes you're using GoDaddy as your domain provider. It should be easy to adapt to other providers.

Live demo📺

I deployed our confidential GitLab to https://gitlab.edgeless.systems/ and synced the Constellation repo from GitHub.

What are the security properties?

Constellation ensures that the entire K8s cluster runs in runtime-encrypted and isolated Confidential VMs (CVMs) and that all data that leaves the CVMs remains encrypted. For example, if Redis or PostgreSQL write data to cloud storage that data is automatically encrypted. The corresponding cryptographic keys are all generated and managed within CVMs. We as DevOps engineers don't need to worry about these. Bottom line is that all data is always encrypted, even during runtime in memory.

Constellation also verifies the integrity and authenticity of all CVMs and that they all are running the same "good" Constellation node image. This way, once constellation init completes, you know that you are actually talking to an end-to-end confidential cluster via kubectl.

You can learn more about these concepts in the Constellation docs.

What about user connections?

As I discussed already in a previous post on confidential Rocket.Chat, users of the service should also have means to verify that the service is confidential and that it is the right instance they're talking to.

For browser-based connections, currently, the only way is to manually check the fingerprint of the TLS certificate. In the case of our demo instance of GitLab (https://gitlab.edgeless.systems/) the SHA-256 fingerprint is CB C2 F9 23 7B 46 66 FC 5B 88 70 D5 3C 73 24 98 B4 EC 1D 12 EC 38 85 8C FA A6 44 B2 43 73 34 18🧐

For Git CLI-based connections, things are simpler. By adding the following entry to our gitconfig file, we can ensure that Git always checks that the right certificate is present when talking to https://gitlab.edgeless.systems/:

[http "https://gitlab.edgeless.systems/"]
    pinnedPubkey = sha256//aZTzzCepU+Sa34+xkqyFGiWXG+/yHtF6Q5AgMMAjHhs=

With this in place, we know for sure that commands like git clone https://gitlab.edgeless.systems/edgelesssys/constellation.git or a corresponding git push will always go to the right confidential instance of GitLab and can't be diverted or intercepted by an attacker in control of the host of gitlab.edgeless.systems or the corresponding DNS entries.

Wrap up

Hope you enjoyed the quick walkthrough. Personally, I have to admit that I love the idea of having a GitLab running on Azure, which can't be accessed from the infrastructure and keeps all of the data always encrypted.