DEV Community

Cover image for Zero-Downtime GraphQL Evolution in GitLab: Fixing Multi-Node Upgrades with @gl_introduced
Manish Kumar
Manish Kumar

Posted on

Zero-Downtime GraphQL Evolution in GitLab: Fixing Multi-Node Upgrades with @gl_introduced

#api #architecture #backend #graphql

A deep dive into GraphQL schema evolution, rolling deployments, mixed-version infrastructure, and backward-compatible frontend queries.

Distributed applications rarely upgrade every server instance simultaneously.

In a multi-node deployment, different application versions can temporarily coexist while a rolling upgrade is in progress. This creates an important compatibility challenge: frontend assets served by a newer application version may send GraphQL requests to an older backend node that does not yet understand the updated schema.

This is not simply a deployment sequencing problem. It is a contract compatibility problem between independently deployed frontend and backend components.

While contributing to GitLab, I investigated and fixed this exact issue on the GitLab Credits user page in GitLab Merge Request !268111, addressing Issue #638245.

The fix uses GitLab's @gl_introduced directive to make newly introduced GraphQL fields and arguments compatible with mixed-version environments during rolling upgrades.

This article walks through the failure mode, the underlying schema compatibility problem, the implementation, and the verification process.

1. The Distributed Systems Problem: Mixed-Version Deployments

Consider a GitLab installation running multiple application nodes behind a load balancer.

Initially, all nodes run GitLab 19.5:

                 Load Balancer
                       |
              +--------+--------+
              |                 |
          Node A              Node B
          GitLab 19.5         GitLab 19.5
Enter fullscreen mode Exit fullscreen mode

Now suppose the deployment begins upgrading to GitLab 19.6.

During a rolling upgrade, Node A is upgraded first while Node B continues running the previous version.

                 Load Balancer
                       |
              +--------+--------+
              |                 |
          Node A              Node B
          GitLab 19.6         GitLab 19.5
              |                 |
              +--------+--------+
                       |
               Shared Application
Enter fullscreen mode Exit fullscreen mode

At this point, both versions are active.

This is an expected intermediate deployment state, not necessarily an infrastructure failure.

However, it introduces a compatibility boundary.

1.1 How the mismatch occurs

Consider the following sequence:

  1. A browser requests the GitLab Credits user page.
  2. The upgraded node serves frontend assets containing the GitLab 19.6 GraphQL queries.
  3. The browser executes one of those queries.
  4. The load balancer routes the GraphQL request to the older GitLab 19.5 node.
  5. The older node validates the query against its previous GraphQL schema.
  6. The schema does not recognize the newly introduced argument.

The frontend and backend are individually functioning, but their contracts are temporarily incompatible.

This is a classic mixed-version deployment problem.

2. Understanding the GraphQL Schema Compatibility Failure

The GitLab Credits user page retrieves usage and subscription information through GraphQL queries.

In GitLab 19.6, the includeNonBillable argument was introduced to the following fields:

  • usedFlowTypes
  • events

The updated frontend queries started supplying this argument.

For example:

usedFlowTypes(includeNonBillable: true) {
  id
  title
}
Enter fullscreen mode Exit fullscreen mode

The newer GraphQL schema understands the argument.

The older schema does not.

2.1 What happens on the older node?

When the query reaches a GitLab 19.5 node, GraphQL validates it against the schema available on that node.

The older schema cannot recognize the newly introduced argument.

The resulting validation errors are:

Field 'usedFlowTypes' doesn't accept argument 'includeNonBillable'

Field 'events' doesn't accept argument 'includeNonBillable'
Enter fullscreen mode Exit fullscreen mode

GraphQL performs schema validation before normal query execution.

Therefore, an unknown argument is not simply ignored as an unsupported optional parameter.

The query fails validation.

This can prevent the affected operation from returning the expected data and break the Credits user page during the upgrade window.

2.2 Why this is more than a frontend issue

The frontend query is valid against the new schema.

The backend query validator is also behaving correctly against its own schema.

The actual problem is that the two components are temporarily operating with different versions of the same API contract.

This distinction matters when debugging distributed applications.

A query can be completely valid in the newest application version while remaining invalid on an older node that is still serving production traffic.

3. GraphQL Schema Evolution During Rolling Upgrades

GraphQL APIs evolve over time.

New fields, arguments, enum values, and input types are introduced as application requirements change.

In a single-version environment, frontend and backend changes can often be deployed together without exposing compatibility problems.

Multi-node deployments are different.

During a rolling upgrade, the system must tolerate requests crossing version boundaries.

A useful way to reason about this is to separate three concepts:

  1. Schema availability: Does the backend version understand the requested field or argument?
  2. Client compatibility: Can the frontend safely communicate with older backend versions?
  3. Deployment compatibility: Can multiple application versions serve requests simultaneously?

A robust rolling deployment needs these properties to remain compatible throughout the transition.

3.1 The problem with introducing arguments directly

Suppose a frontend query changes from:

usedFlowTypes {
  id
  title
}
Enter fullscreen mode Exit fullscreen mode

To:

usedFlowTypes(includeNonBillable: true) {
  id
  title
}
Enter fullscreen mode Exit fullscreen mode

The new query depends on an argument that does not exist in the older schema.

If the browser sends this query to an older node, validation fails.

The important point is that the compatibility problem exists even though the new argument is meaningful and supported in the upgraded application.

The frontend needs a mechanism to account for the version of the backend receiving the request.

4. Understanding GitLab's @gl_introduced Directive

GitLab provides version-aware GraphQL compatibility support through the @gl_introduced directive.

It allows frontend GraphQL operations to declare when a field or selection was introduced.

For example:

usedFlowTypes(includeNonBillable: true)
  @gl_introduced(version: "19.6.0") {
  id
  title
}
Enter fullscreen mode Exit fullscreen mode

The annotation communicates that this selection belongs to a schema capability introduced in GitLab 19.6.0.

GitLab's multi-version compatibility layer can use this version information to omit the annotated selection when communicating with an older backend version that does not support it.

This prevents the older backend from receiving an incompatible field or argument as part of the transformed query.

4.1 Why the directive matters

Without version metadata, the compatibility layer cannot reliably distinguish a newly introduced selection from one that is expected to exist across supported versions.

The directive provides that information explicitly.

Conceptually, the process looks like this:

             Frontend GraphQL Query
                       |
                       v
             Version Compatibility
                    Layer
                       |
             +---------+---------+
             |                   |
             v                   v
       GitLab 19.6           GitLab 19.5
       New Schema            Old Schema
             |                   |
             v                   v
       Keep New Field       Omit New Selection
             |                   |
             v                   v
       Execute Query        Execute Compatible
                            Query
Enter fullscreen mode Exit fullscreen mode

The important distinction is that the directive is compatibility metadata used by GitLab's query-processing pipeline. It is not a native GraphQL mechanism that automatically changes server schemas.

4.2 Compatibility is based on introduction version

The version annotation must accurately reflect when the relevant schema capability became available.

For this contribution, the introduction version was:

@gl_introduced(version: "19.6.0")
Enter fullscreen mode Exit fullscreen mode

Using the correct version allows the compatibility mechanism to distinguish GitLab 19.5 from GitLab 19.6.

This is particularly important during the transition period when both versions can serve requests.

5. Implementing the Fix

The fix was applied to two GraphQL query files in the GitLab Credits usage billing frontend.

5.1 Updating the subscription usage query

File:

ee/app/assets/javascripts/usage_quotas/usage_billing/users/show/graphql/get_user_subscription_usage.query.graphql
Enter fullscreen mode Exit fullscreen mode

The usedFlowTypes selection was updated to include the introduction directive:

usedFlowTypes(includeNonBillable: true)
  @gl_introduced(version: "19.6.0") {
  id
  title
}
Enter fullscreen mode Exit fullscreen mode

This marks the selection as introduced in GitLab 19.6.

When the compatibility layer communicates with an older supported node, it can omit the version-gated selection instead of sending an argument that the older schema does not recognize.

5.2 Updating the subscription usage events query

The second change was made in:

ee/app/assets/javascripts/usage_quotas/usage_billing/users/show/graphql/get_user_subscription_usage_events.query.graphql
Enter fullscreen mode Exit fullscreen mode

The events selection was updated similarly:

events(
  after: $after
  before: $before
  includeNonBillable: true
) @gl_introduced(version: "19.6.0") {
  # Existing selected fields
}
Enter fullscreen mode Exit fullscreen mode

This ensures that the newly introduced includeNonBillable argument is covered by the version compatibility annotation.

5.3 Why both locations needed changes

The issue involved two separate GraphQL selections.

Fixing only usedFlowTypes would leave the events query vulnerable to the same schema validation failure.

Likewise, annotating only events would not protect the subscription usage query.

Both selections needed the same introduction version because both depended on the GitLab 19.6 schema change.

The resulting implementation was deliberately focused:

  • No changes to the backend schema.
  • No changes to the underlying business logic.
  • No changes to the load-balancing strategy.
  • No need to introduce a separate frontend version-detection mechanism.

Instead, the existing GraphQL compatibility infrastructure was used to describe the version dependency directly at the query level.

6. Verification and Regression Testing

After updating both GraphQL queries, I ran the usage billing frontend Jest suite.

Command:

yarn jest ee/spec/frontend/usage_quotas/usage_billing/
Enter fullscreen mode Exit fullscreen mode

Verified result:

10 test suites passed
165 tests passed
0 failures
Enter fullscreen mode Exit fullscreen mode

This verified that the existing usage billing frontend tests continued to pass after the query updates.

6.1 Why regression testing matters

GraphQL query files are part of the frontend application's API contract.

Even a small annotation change can affect how query selections are processed.

Running the relevant test suite helps verify that the changes do not introduce regressions in the surrounding usage billing functionality.

The verification was focused on the affected frontend area rather than relying only on the visual behavior of the Credits user page.

7. Engineering Considerations for Multi-Node Deployments

This contribution highlights several broader principles that are important when evolving APIs in distributed applications.

7.1 Deployments are not always atomic

In multi-node systems, application versions can coexist temporarily.

Any frontend change that depends on a newly introduced backend capability should account for this transition period.

Assuming that every request reaches the newest server version is unsafe in a rolling deployment.

7.2 GraphQL validation happens before normal execution

An unknown argument can invalidate an operation before its resolver executes.

This is why an optional-looking argument is not automatically backward compatible.

Compatibility needs to be handled before the query reaches a schema that cannot validate it.

7.3 Version metadata makes compatibility explicit

The @gl_introduced directive provides a clear declaration of when a GraphQL selection became available.

This is more maintainable than scattering assumptions about backend versions across application code.

7.4 Fix compatibility at the correct layer

The underlying schema change was intentional.

The problem was not that GitLab 19.6 introduced a new argument.

The problem was that the frontend needed to communicate safely with older nodes during the upgrade process.

Using the existing version compatibility mechanism addressed the actual failure without introducing unnecessary backend or infrastructure changes.

8. Final Summary

In GitLab MR !268111, I fixed a GraphQL compatibility problem affecting the GitLab Credits user page during multi-node rolling upgrades.

The failure occurred because frontend assets from GitLab 19.6 could send queries containing the newly introduced includeNonBillable argument to GitLab 19.5 nodes.

Those older nodes rejected the queries during GraphQL schema validation.

I added:

@gl_introduced(version: "19.6.0")
Enter fullscreen mode Exit fullscreen mode

to both affected selections:

  • usedFlowTypes
  • events

This allows GitLab's multi-version compatibility layer to handle the new selections appropriately when communicating with older application versions.

Verification results:

  • 10 Jest test suites passed.
  • 165 tests passed.
  • 0 failures.

The broader engineering takeaway is that API evolution in distributed systems must account for deployment topology, not just the latest application version.

A frontend query that works perfectly against the newest schema may still fail in production when requests are routed to older nodes.

Version-aware GraphQL compatibility is an important part of making rolling deployments reliable.

References and Implementation

  • Issue: gitlab-org/gitlab#638245
  • Merge Request: gitlab-org/gitlab!268111
  • Subscription Usage Query: get_user_subscription_usage.query.graphql
  • Subscription Usage Events Query: get_user_subscription_usage_events.query.graphql
  • Test Directory: ee/spec/frontend/usage_quotas/usage_billing/

Author: Manish Kumar

GitLab: @manishpatel00

Contributing to GitLab's GraphQL architecture, frontend infrastructure, and distributed application reliability.

Top comments (0)