Use minio as an external object storage with Gitlab

Context

Gitlab can be installed on a Kubernetes cluster with the help of the official Gitlab Helm chart. Here I install it on a Rancher managed Kubernetes cluster with the usual griefs that come with the installation of such a behemoth: optimizing resources, picking the right subchart to install and how to install it. The installation is notoriously tedious given the substantial amount of subcharts and options, not counting the optimization needed to fit it in a resource-constrained cluster. Everything was working fine until we had a cluster issue with nodes crashing and I had to reinstall Gitlab from the custom chart I created. The installation only kept the gitaly, postgresql and redis PVCs, and to my devopsy sorrow, the minio storage was gone. There is no way around it, minio has to be installed seprately for a production ready self managed Gitlab. Here's how.

Pre-requisites

• A working Kubernetes installation and a distributed block storage for Kubernetes (I am using Longhorn on Rancher)
• A working helm tooling
• Needles to say, a sufficiently provisioned cluster with preferably a backup and restoration routine

Install a minio chart

We'll use a Bitnami chart to install a standalone Minio instance:

minio-external:
  mode: standalone
  enabled: true
  persistence:
    enabled: true
    size: 10Gi
    annotations:
      helm.sh/resource-policy: keep
  auth:
    rootUser: <REDACTED>
    rootPassword: <REDACTED>
  provisioning:
    enabled: true
    users:
      - username: <REDACTED>
        password: <REDACTED>
        disabled: false
        policies:
          - readwrite
          - consoleAdmin
          - diagnostics  
        setPolicies: false
  accessKey:
    password: <REDACTED>
  secretKey:
    password: <REDACTED>
  # volumePermissions:
  #   enabled: true
  defaultBuckets: default,gitlab-registry-storage,gitlab-lfs,gitlab-artifacts,gitlab-uploads,gitlab-packages,gitlab-mr-diffs,gitlab-terraform-state,gitlab-ci-secure-files,gitlab-dependency-proxy,gitlab-pages  

Use this configuration to kickstart a Minio instance with the buckets needed by Gitlab (defaultBuckets). These buckets are needed by different services of Gitlab and they will be mentionned in appConfig later.

The default minio values for this helm chart are available [here](# charts/bitnami/minio/values.yaml at main · bitnami/charts · GitHub).

Disable managed minio

Since we are using an external Minio instance we do not need the Gitlab managed one anymore. In your Gitlab values file disable minio:

global:
  ...
  minio:
    enabled: false

⚠️ This will delete your Gitlab Minio instance and all its data. Please proceed with caution. Consider backuping your data and migrating it afterwards using your tool of choice. I would use minio command line.

Create a secret

apiVersion: v1  
kind: Secret
metadata:  
    name: gitlab-object-storage
stringData:
  connection: |
    provider: AWS
    region: us-east-1
    aws_access_key_id: <REDACTED>
    aws_secret_access_key: <REDACTED>
    endpoint: "gitlab-minio-external:9000"  

Connect Gitlab to the new Minio instance

  global:
    registry:
      bucket: gitlab-registry-storage
    appConfig: 
      ...  
      object_store:
        enabled: true
        proxy_download: true
        storage_options:
          {}
          # server_side_encryption:
          # server_side_encryption_kms_key_id
        connection:
          secret: gitlab-object-storage
          key: connection
      lfs:
        enabled: true
        proxy_download: true
        bucket: gitlab-lfs   
      artifacts:
        enabled: true
        proxy_download: true
        bucket: gitlab-artifacts
      uploads:
        enabled: true
        proxy_download: true
        bucket: gitlab-uploads
      packages:
        enabled: true
        proxy_download: true
        bucket: gitlab-packages
      externalDiffs:
        enabled: true
        when:
        proxy_download: true
        bucket: gitlab-mr-diffs
      terraformState:
        enabled: true
        bucket: gitlab-terraform-state
      ciSecureFiles:
        enabled: true   
        bucket: gitlab-ci-secure-files
        # connection:
        #   secret: gitlab-object-storage
      dependencyProxy:
        enabled: true
        proxy_download: true
        bucket: gitlab-dependency-proxy
      pages:
        enabled: true
        proxy_download: true
        bucket: gitlab-pages

Launch your gitlab installation and it should work.

Voilà!

Inspirations and references

• https://docs.gitlab.com/charts/advanced/external-object-storage/
• https://docs.gitlab.com/charts/advanced/external-object-storage/
• https://docs-bigbang.dso.mil/latest/packages/gitlab/docs/operational-production-settings/
• https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1039
• https://www.aidoos.com/kb/devops-gitlab-configure-minio-with-the/
• Connection secret https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/charts/globals.md#connection
• https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/objectstorage/registry.minio.yaml
• https://gitlab.com/gitlab-org/charts/gitlab/blob/master/examples/values-external-objectstorage.yaml
• https://forum.gitlab.com/t/user-uploads-to-s3-buckets-are-invalid/69054/4
• https://gitlab.com/gitlab-org/charts/gitlab/-/issues/4003