Progressive cluster upgrades at scale: A technical guide to GKE rollout sequencing with custom stages

Upgrading Kubernetes clusters across a large enterprise fleet is often a balancing act between staying current with security patches and avoiding outages. By default, Google Kubernetes Engine (GKE) rolls out automatic upgrades progressively according to Google Cloud regional timelines. While regional rollout works well for standalone clusters, it does not understand your organization's business topology. If you run staging clusters in us-central1 and critical production clusters in us-east1, a standard regional rollout could upgrade your production environment before your pre-production validation completes.

The General Availability (GA) release of GKE rollout sequencing with custom stages solves this challenge. It provides platform teams with declarative control to sequence cluster upgrades across fleets, environments, and even distinct Google Cloud organizations according to business criticality rather than cloud geography.

How rollout sequencing works

Rollout sequencing builds on GKE fleet management. Fleets serve as logical boundaries for environments such as development, staging, and production. With rollout sequencing, you define an ordered pipeline of upgrade stages managed by a central resource called RolloutSequence.

When GKE publishes a new automatic upgrade target for a release channel, or when you explicitly trigger a target version, the system creates a Rollout object. This rollout progresses through your defined stages sequentially:

• Control plane upgrades start in the first stage. Once all control planes in that stage reach the target version, a stage soak timer begins.
• Node upgrades run in parallel with control plane upgrades, respecting node pool upgrade strategies such as surge or blue-green.
• When both control planes and nodes complete their upgrade and satisfy the configured soak duration, the rollout advances to the next stage in the sequence.

If an individual stage contains clusters that take longer than 30 days to finish upgrading—due to restrictive maintenance windows or exclusions—GKE triggers a forced soak period to avoid stalling the entire multi-stage pipeline indefinitely.

Granular stages with label selectors

Earlier fleet-based rollout sequencing operated strictly at the fleet level, meaning an entire fleet had to upgrade before another fleet could begin. Custom stages introduce the ability to split a single fleet into multiple granular rollout phases using Common Expression Language (CEL) label selectors.

For example, within a production fleet, you can label a subset of clusters as canary targets and upgrade them before the rest of production.

Here is an example YAML manifest defining a three-stage sequence:

- stage:
  fleet-projects:
  - projects/dev-fleet-host
  soak-duration: 3d
- stage:
  fleet-projects:
  - projects/prod-fleet-host
  label-selector: resource.labels.tier=='canary'
  soak-duration: 4d
- stage:
  fleet-projects:
  - projects/prod-fleet-host
  soak-duration: 7d

When structuring custom stages, keep two architectural rules in mind:

• Catch-all requirement: If you use a label selector to target a subset of clusters in a fleet, the final stage referencing that fleet must omit the label selector. This acts as a catch-all stage to guarantee that all remaining clusters in the fleet are upgraded.
• Conflict resolution: If a cluster matches multiple stages within a sequence, GKE assigns that cluster exclusively to the earliest matching stage.

You can register this configuration using the Google Cloud CLI:

gcloud container fleet rolloutsequences create prod-rollout-sequence \
    --display-name="Production rollout sequence" \
    --stage-config=rollout-sequence.yaml \
    --project=central-management-project

Operational controls and upgrade scoping

Real-world production environments require operational agility when unexpected workload anomalies occur. Rollout sequencing with custom stages provides real-time lifecycle controls over active rollouts without requiring you to dismantle your configuration:

• Pause and resume: If an issue arises during validation in a canary stage, you can pause the active rollout. Pausing stops GKE from initiating new cluster upgrades in that stage and subsequent stages while allowing running operations to finish. Once mitigated, you can resume the rollout.
• Stage completion: If automated testing confirms stability before a soak timer finishes, platform operators can execute a stage completion action (force-complete-stage). This skips the remaining soak duration and moves the rollout immediately to the next stage.
• Rollout cancellation: If a version introduces an unresolvable regression, you can cancel the rollout. Canceling stops the progression of that specific target version across the sequence.

In addition to runtime actions, you can restrict the automatic upgrade scope for a rollout sequence. If your platform policy requires manual control over major and minor Kubernetes version changes while automating security patches, you can configure your sequence to roll out only control plane patch upgrades and node patch upgrades.

However, mandatory upgrades—such as control planes that have not received a patch in 90 days or clusters reaching the end of version support—will continue to execute to preserve cluster stability and security.

Multi-tenant and cross-organization management

For enterprises with distributed topologies, rollout sequences support cross-project and cross-organization fleet memberships. Following continuous delivery best practices, Google recommends creating and maintaining RolloutSequence resources in a dedicated host project.

A single rollout sequence can define up to 15 distinct stages, and fleets can accommodate up to 250 clusters (or up to 2,000 clusters when using lightweight memberships with an approved quota increase). Across all stages in a sequence, you can configure a total soak duration of up to 90 days, with individual stage soak times configured up to 30 days.

By shifting from regional upgrade schedules to business-aligned rollout sequences, platform engineers can automate Kubernetes lifecycle management while maintaining safety, visibility, and control across their entire fleet.

To implement progressive cluster upgrades in your environment, explore the official GKE rollout sequencing documentation to review detailed configuration parameters, CEL filter syntax, and gcloud CLI commands.