DEV Community

Cover image for How I Fixed GitLab's Self-Hosted Observability Architecture and Earned Level 3 Contributor Status
Manish Kumar
Manish Kumar

Posted on

How I Fixed GitLab's Self-Hosted Observability Architecture and Earned Level 3 Contributor Status

#architecture #debugging #opensource #ruby

A deep dive into debugging SaaS-centric assumptions in enterprise codebases, improving self-hosted observability, navigating GitLab's CI pipelines, and contributing to one of the world's largest open-source Ruby on Rails applications.

Executive Summary

A small assumption in URL parsing can have consequences across an entire application. In this contribution, I fixed an observability configuration issue where self-hosted service URLs were incorrectly treated as GitLab SaaS tenant identifiers.

The work involved more than changing a Ruby method. It required correcting endpoint resolution, protecting frontend templates against missing configuration, updating tests and translations, and working through GitLab's review and CI requirements.

1. Introduction: The Hidden SaaS Assumption in Observability

Contributing to GitLab means working in a codebase with years of accumulated engineering decisions, multiple deployment models, and many interconnected components.

One recent issue I investigated was GitLab work item #638990, which exposed an architectural assumption in GitLab Observability, commonly referred to as O11y.

GitLab Observability allows teams to send OpenTelemetry (OTel) telemetry, including traces, metrics, and logs, to an observability backend.

For GitLab.com, tenant-specific observability services use hostnames under:

*.gitlab-o11y.com
Enter fullscreen mode Exit fullscreen mode

However, self-managed GitLab administrators may configure their own observability services, for example:

https://o11y.internal.corp.com
Enter fullscreen mode Exit fullscreen mode

The application was deriving a service name as though every configured URL belonged to GitLab's hosted observability domain.

That assumption caused self-hosted URLs to be interpreted incorrectly, which affected the default endpoint configuration.

The central engineering question was:

How can GitLab preserve its existing SaaS endpoint behavior while correctly supporting custom self-hosted observability URLs?

2. Root Cause Analysis: The Danger of Naive Domain Stripping

The issue originated in:

app/models/observability/group_o11y_setting.rb
Enter fullscreen mode Exit fullscreen mode

The Observability::GroupO11ySetting model derived an O11y service name from the configured service URL.

The original implementation

The previous logic used string operations:

def name_from_url
  return unless o11y_service_url.present?

  o11y_service_url
    .delete_prefix('https://')
    .delete_suffix('.gitlab-o11y.com')
end
Enter fullscreen mode Exit fullscreen mode

This implementation implicitly assumed that the configured URL always followed the GitLab-hosted domain convention:

https://<tenant>.gitlab-o11y.com
Enter fullscreen mode Exit fullscreen mode

What happened with a self-hosted URL?

Consider this configuration:

https://o11y.internal.corp.com
Enter fullscreen mode Exit fullscreen mode

The operations produce:

'https://o11y.internal.corp.com'
  .delete_prefix('https://')
  .delete_suffix('.gitlab-o11y.com')
Enter fullscreen mode Exit fullscreen mode

The result is:

o11y.internal.corp.com
Enter fullscreen mode Exit fullscreen mode

Because the hostname does not end with .gitlab-o11y.com, the suffix removal does nothing.

The model then treats the complete self-hosted hostname as a short service name.

How the incorrect name affected endpoints

The model also constructed default OTel endpoints using the derived service name:

def default_otel_url
  "https://#{o11y_service_name}.otel.gitlab-o11y.com:14318"
end
Enter fullscreen mode Exit fullscreen mode

For the self-hosted configuration, this could produce:

https://o11y.internal.corp.com.otel.gitlab-o11y.com:14318
Enter fullscreen mode Exit fullscreen mode

That is not the administrator's configured endpoint. It is an incorrectly constructed GitLab-hosted hostname.

The problem was therefore not just an incorrect display value. It affected the endpoint resolution path used by observability configuration.

The wider impact

The issue had consequences in multiple areas:

Incorrect endpoint resolution

Self-hosted deployments could receive generated SaaS-style endpoints instead of using their custom observability configuration.

Frontend quick-start failures

When OTel endpoints were absent or invalid, the quick-start configuration in _endpoint_configuration.html.haml could render incomplete data for QuickStartSnippets.vue, resulting in frontend JavaScript errors.

Confusing administration UI

The GitLab Admin Panel displayed a column labeled Service Name, even when the configured value represented a complete self-hosted service URL.

The underlying problem was a mismatch between two deployment models: GitLab-hosted services and administrator-managed services.

3. The Engineering Solution

The fix required changes across the model, endpoint resolution, frontend templates, tests, and documentation.

The implementation followed five principles:

  1. Identify GitLab-hosted observability URLs explicitly.
  2. Avoid generating SaaS defaults for self-hosted configurations.
  3. Handle missing endpoint values safely.
  4. Render frontend quick-start instructions only when the required configuration exists.
  5. Document the self-managed configuration behavior.

A. Replace String Slicing with Explicit Host Validation

Instead of assuming that every URL belongs to GitLab's hosted domain, the updated implementation parses the URL and validates its hostname.

The matching pattern is:

GITLAB_O11Y_HOST_REGEX = /\A([a-z0-9-]+)\.gitlab-o11y\.com\z/i
Enter fullscreen mode Exit fullscreen mode

The revised method follows this approach:

def name_from_url
  return unless o11y_service_url.present?

  host = Addressable::URI.parse(o11y_service_url).host
  match = GITLAB_O11Y_HOST_REGEX.match(host)

  match[1] if match
rescue Addressable::URI::InvalidURIError
  nil
end
Enter fullscreen mode Exit fullscreen mode

The important difference is that the service name is extracted only when the hostname matches the expected GitLab-hosted domain.

For example:

Configured URL Derived hosted service name
https://tenant.gitlab-o11y.com tenant
https://o11y.internal.corp.com nil
Invalid URI nil

When the URL does not represent a recognized GitLab-hosted service, name_from_url returns nil.

The model can then fall back to its group-derived name through name_from_group.

This is more reliable than trying to infer a tenant identifier by removing fixed strings from an arbitrary URL.

B. Guard Default OTel and MCP Endpoints

Correctly identifying the service name is only one part of the solution.

The application must also avoid generating GitLab SaaS endpoints for self-hosted configurations.

The endpoint methods were updated to use configured values first and generate defaults only for GitLab-hosted services:

def o11y_otel_url
  super.presence || (default_otel_url if gitlab_hosted?)
end

def o11y_mcp_url
  super.presence || (default_mcp_url if gitlab_hosted?)
end
Enter fullscreen mode Exit fullscreen mode

The hosted-service check is encapsulated in a private method:

private

def gitlab_hosted?
  self.class.hosted_service? || name_from_url.present?
end
Enter fullscreen mode Exit fullscreen mode

This separates two responsibilities:

  • Use an explicitly configured endpoint when one is present.
  • Generate the default endpoint only when the service belongs to the GitLab-hosted observability configuration.

For self-hosted installations, this prevents the model from inventing a SaaS endpoint based on an unrelated hostname.

It also keeps the existing GitLab-hosted behavior intact.

C. Make Endpoint Helpers and Frontend Rendering Safe

The backend endpoint changes also required the frontend to handle missing configuration gracefully.

The relevant template was:

app/views/shared/observability/setup/_endpoint_configuration.html.haml
Enter fullscreen mode Exit fullscreen mode

Previously, the quick-start snippets and associated curl examples could be rendered even when the OTel HTTPS endpoint was unavailable.

The template was updated to render those instructions only when the endpoint exists:

- if otel_https_endpoint.present?
  #js-observability-quick-start{ data: { ... } }
- else
  .gl-alert.gl-alert-tip
    = s_('Observability|Configure an OpenTelemetry endpoint above to view quick-start instructions.')
Enter fullscreen mode Exit fullscreen mode

This gives administrators a clear configuration message instead of rendering incomplete quick-start instructions.

The principle is simple: frontend components should not assume that optional backend configuration is always present.

When a required endpoint is missing, the UI should communicate what needs to be configured.

D. Keep the Documentation Aligned

The implementation also involved updating:

doc/operations/observability/setup_self_managed.md
Enter fullscreen mode Exit fullscreen mode

Self-managed observability documentation needs to explain how administrators configure their own endpoints and how those endpoints interact with GitLab's observability setup.

The documentation review helped ensure that the behavior described for self-managed deployments matched the actual implementation.

4. Navigating GitLab's CI and Review Process

Implementing the fix was only part of the work. Getting it ready for a large production codebase required addressing several testing, static-analysis, and review requirements.

These were some of the most important engineering challenges.

A. The let_it_be Mutation Trap in RSpec

GitLab uses let_it_be to create test records once and reuse them across examples. This improves test performance, but shared records are protected against accidental mutation.

During controller update tests, modifying a shared frozen configuration object could produce:

FrozenError: can't modify frozen Hash
Enter fullscreen mode Exit fullscreen mode

The solution was to use:

let_it_be_with_reload(:o11y_setting) do
  create(:observability_group_o11y_setting)
end
Enter fullscreen mode Exit fullscreen mode

This allowed the test to reload the record between examples while retaining the performance benefits of shared test setup.

The broader lesson is that test setup strategies matter. A fixture designed for read-only reuse is not necessarily appropriate when examples mutate the underlying record.

B. Undercover Code Coverage Enforcement

GitLab uses Undercover to enforce test coverage for changed code.

Adding an explicit rescue path for malformed URLs meant that the new behavior needed its own test coverage.

For example:

context 'with invalid URI' do
  let(:url) { 'http://[' }

  it { is_expected.to be_nil }
end
Enter fullscreen mode Exit fullscreen mode

This verifies that invalid URI input follows the intended fallback path instead of allowing a parsing exception to escape.

The important lesson was to test error-handling branches deliberately, not just the normal URL parsing path.

C. Gettext Translation File Drift

GitLab uses Gettext for translatable strings.

When changing strings such as:

s_('Observability|Configure an OpenTelemetry endpoint above to view quick-start instructions.')
Enter fullscreen mode Exit fullscreen mode

the corresponding translation template also needs to be regenerated.

The command used was:

bin/rake gettext:regenerate
Enter fullscreen mode Exit fullscreen mode

The generated changes in:

locale/gitlab.pot
Enter fullscreen mode Exit fullscreen mode

must remain synchronized with the source strings.

Otherwise, static-analysis checks can report translation inconsistencies.

This was a useful reminder that changing a user-facing message in GitLab often involves more than editing a view template.

D. Working Through GitLab Duo and Senior Engineering Review

The review process involved both automated feedback and human reviewers.

GitLab Duo

The AI review process helped identify edge cases around empty endpoints and translation consistency.

Jorge Tomás — @jotolo_gl

The review feedback helped refine the encapsulation of gitlab_hosted? into private scope and encouraged testing through externally observable behavior, such as the endpoint methods, rather than coupling tests to internal implementation details.

Roshni Sarangadharan — @rsarangadharan

The technical writing review helped ensure that the self-managed observability documentation accurately described the configuration behavior.

This process reinforced an important point: a successful open-source contribution is not only about writing code. It also involves explaining the design, responding to review feedback, and making the implementation maintainable for future contributors.

5. The Outcome: Approval and Level 3 Contributor Status

After addressing the review feedback, updating the implementation and tests, and pushing the final commits, the merge request reached approval.

The milestone included:

  • Approval from staff engineering and technical writing reviewers.
  • A green CI pipeline.
  • More than 88 CI jobs passing, as recorded during the contribution.
  • 150 GitLab reward credits.
  • Advancement to GitLab Level 3 Community Contributor.

GitLab also sent a contributor reward notification:

You've received 150 reward credits from GitLab!

Congratulations @manishpatel00!

Your contributions to GitLab have made a real impact on our open source community!

For me, this milestone represented more than a reward. It reflected the work involved in understanding an unfamiliar part of a large codebase, identifying an architectural assumption, implementing a safe fix, and taking the contribution through review and CI.

6. Key Engineering Takeaways

1. Never assume SaaS and self-hosted environments share the same domain structure

Products that support both hosted and self-managed deployments need explicit boundaries between their configuration models.

A hostname convention that works for a SaaS tenant should not automatically be applied to an arbitrary administrator-provided URL.

2. Prefer parsing and validating URLs over manipulating strings

Removing prefixes and suffixes is not equivalent to understanding a URL.

Parsing the hostname and validating it against an explicit domain pattern makes the intended behavior clearer and reduces accidental matches.

3. Missing configuration should produce a safe, understandable state

When an endpoint is unavailable, the application should avoid constructing invalid defaults or rendering incomplete instructions.

A clear configuration message is better than a broken quick-start experience.

4. Test edge cases and failure paths

Valid hosted URLs are only one part of the behavior.

Self-hosted URLs, missing values, malformed URLs, and fallback behavior all deserve explicit coverage.

5. Treat CI and code review as part of engineering

Undercover, RSpec, Gettext, static analysis, and human review are not administrative steps after implementation.

They help establish that a change is correct, understandable, and safe to maintain in a large codebase.

6. Review feedback is an opportunity to improve design

The discussions around private method scope, external behavior testing, and documentation accuracy helped strengthen the contribution beyond the original bug fix.

7. Final Thoughts

This contribution started with a seemingly small URL parsing issue.

But the root cause exposed a broader architectural assumption: GitLab's observability service-name derivation was treating self-hosted URLs as if they belonged to the GitLab SaaS domain.

Fixing it required explicit hostname validation, safer endpoint resolution, defensive frontend rendering, test coverage for edge cases, translation updates, and accurate self-managed documentation.

The experience also demonstrated what contributing to a large open-source project really involves. The code change is only one part of the journey. Understanding existing architecture, navigating CI, responding to reviewers, and documenting behavior are equally important.

I'm glad to have contributed this fix to GitLab and to have reached Level 3 Community Contributor status, along with 150 reward credits.

If you're contributing to a large open-source project, don't underestimate small bugs. Sometimes a single incorrect assumption reveals an important architectural boundary worth fixing.

References

Author: Manish Kumar

GitLab: @manishpatel00

Top comments (0)