DEV Community

Pavan Kumar Sadashiv
Pavan Kumar Sadashiv

Posted on

Solving x509 Self-Signed Certificate Errors in Dockerized GitLab & Mattermost

#docker #gitlab #devops #security

Solving x509 Self-Signed Certificate Errors in Dockerized GitLab & Mattermost

If you have ever deployed self-hosted GitLab alongside Mattermost using Docker or Omnibus in an enterprise intranet or air-gapped environment, you have almost certainly encountered this dreaded error:

x509: certificate signed by unknown authority
Enter fullscreen mode Exit fullscreen mode

Everything seems to work fine when testing with curl -k or opening the web browser with an accepted certificate warning. But the moment Mattermost tries to authenticate against GitLab's OAuth endpoint or send an internal webhook, the connection abruptly fails.

Recently, I contributed Merge Request !9851 to Omnibus GitLab (resolving Issue #841) to address and document how custom CA certificates and Docker container isolation interact in production environments.

In this article, we’ll break down:

  1. Why Go services (like Mattermost) fail to validate self-signed certificates inside Docker.
  2. The difference between host trust stores and container CA bundles.
  3. The exact configuration steps to permanently fix the issue in docker-compose.yml and gitlab.rb.
  4. How to contribute documentation fixes to Omnibus GitLab while passing Vale, Lychee, and Markdownlint.

1. The Root Cause: Why Go Rejects Your Self-Signed Cert

Mattermost is written in Go. Unlike languages or tools that rely on OpenSSL environment variables, Go’s standard library (crypto/x509) reads the system’s Root CA bundle from specific operating system paths at startup.

When running inside a Docker container:

  1. Container Isolation: Installing your internal Root CA certificate on the host machine (/usr/local/share/ca-certificates/) does not automatically propagate into running Docker containers.
  2. Missing Trust Store Update: If you only mount the certificate into /etc/gitlab/trusted-certs/, Omnibus GitLab's internal reconfigure script generates the OpenSSL hash symlinks, but secondary daemons (or separate containers) may still lack access to the compiled system CA store.
  3. Strict Validation: Go strictly checks the entire chain of trust:
    • Is the certificate signed by a known CA?
    • Is the Subject Alternative Name (SAN) matching the container’s internal hostname?
    • Has the certificate expired?

If the root certificate is missing from the container's bundle, Go terminates the TLS handshake immediately with x509: certificate signed by unknown authority.

2. The Solution: Step-by-Step Configuration

To resolve this reliably in a Dockerized GitLab + Mattermost stack, follow this configuration recipe:

Step A: Place the Root CA Certificate on the Host

Create a dedicated certificates directory on your Docker host:

mkdir -p /srv/gitlab/config/trusted-certs
cp my-company-root-ca.crt /srv/gitlab/config/trusted-certs/
Enter fullscreen mode Exit fullscreen mode

Crucial Rule: The file must have a .crt extension (not .pem or .cer), or Omnibus will ignore it during hash indexing.

Step B: Mount Trusted Certs into the Containers

In your docker-compose.yml, mount the trusted certificates volume into both the GitLab container and the Mattermost container:

version: '3.8'

services:
  gitlab:
    image: 'gitlab/gitlab-ce:latest'
    restart: always
    hostname: 'gitlab.local'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.local'
        mattermost_external_url 'https://mattermost.local'
        mattermost['enable'] = true
    ports:
      - '80:80'
      - '443:443'
      - '22:22'
    volumes:
      - '/srv/gitlab/config:/etc/gitlab'
      - '/srv/gitlab/logs:/var/log/gitlab'
      - '/srv/gitlab/data:/var/opt/gitlab'
      # Mount trusted certificates here:
      - '/srv/gitlab/config/trusted-certs:/etc/gitlab/trusted-certs:ro'

  mattermost:
    image: 'mattermost/mattermost-team-edition:latest'
    restart: always
    volumes:
      # If running Mattermost as a standalone container:
      - '/srv/gitlab/config/trusted-certs/my-company-root-ca.crt:/etc/ssl/certs/my-company-root-ca.pem:ro'
Enter fullscreen mode Exit fullscreen mode

Step C: Trigger Certificate Re-hashing

Inside the GitLab container, Omnibus uses c_rehash to link certificates. Run:

docker exec -it gitlab gitlab-ctl reconfigure
Enter fullscreen mode Exit fullscreen mode

When Omnibus reconfigures, it outputs:

Recipe: gitlab::add_trusted_certs
  * directory[/opt/gitlab/embedded/ssl/certs] action create (up to date)
  * file[/opt/gitlab/embedded/ssl/certs/my-company-root-ca.crt] action create
  * execute[/opt/gitlab/embedded/bin/c_rehash /opt/gitlab/embedded/ssl/certs] action run
Enter fullscreen mode Exit fullscreen mode

Step D: Verifying the TLS Handshake Inside the Container

Before testing through the UI, verify the connection from inside the container:

docker exec -it gitlab /opt/gitlab/embedded/bin/openssl s_client \
  -CAfile /opt/gitlab/embedded/ssl/certs/cacert.pem \
  -connect gitlab.local:443
Enter fullscreen mode Exit fullscreen mode

You should see:

Verify return code: 0 (ok)
Enter fullscreen mode Exit fullscreen mode

If it returns code 0, Mattermost and GitLab will now establish encrypted, verified TLS handshakes with zero errors.

3. Contributing Documentation to Omnibus GitLab

Clear documentation is just as valuable as code. In Merge Request !9851, we submitted clear instructions to the official GitLab documentation repository to save hundreds of administrators from debugging this issue.

When you contribute to GitLab documentation (gitlab-org/omnibus-gitlab or gitlab-org/gitlab), your merge request must pass a stringent automated documentation pipeline:

  1. Vale: An open-source linter for technical writing. It enforces the GitLab Style Guide, flagging passive voice, overly complex words, or inconsistent terminology.
  2. Markdownlint: Enforces strict markdown spacing, heading increments, and list indentation.
  3. Lychee: A link checker that verifies every internal and external hyperlink resolves to an active HTTP 200 response.
  4. Hugo: Compiles the entire documentation site to detect formatting or routing breaks.

Pro-Tip: Running Linters Locally Before Pushing 

# Run Vale locally to check your prose
vale doc/

# Run markdownlint
markdownlint-cli2 "doc/**/*.md"
Enter fullscreen mode Exit fullscreen mode

Passing these checks on the first try ensures rapid maintainer approval!

4. Key Takeaways

  1. Go binaries don't use OS cert stores dynamically: They read them at container startup.
  2. Always mount into /etc/gitlab/trusted-certs with .crt extension: Omnibus handles c_rehash automatically.
  3. Docs matter: If you solved a thorny infrastructure issue, open an MR to document it! It helps the entire community and earns recognition on the GitLab Contributor Platform.

Links & References

Have you encountered certificate bugs in Docker or Kubernetes? Share your setup tips in the comments below!

Top comments (0)