Kahiro Okina

Posted on Aug 24

How to Update Open Cluster Management Add-ons in Order: dev stg prod

#kubernetes

By combining ProgressivePerGroup with Placement decision groups, you can roll out add-on configuration changes in the order dev → stg → prod. In this article, I use cluster-proxy as an example to explain the required configuration and how the rollout actually works.

Overview

flowchart TB
    Upgrade["helm upgrade<br/>change tag to vX.Y.Z"]

    subgraph Hub["Hub cluster"]
        direction TB
        Manager["cluster-proxy-addon-manager<br/>update Deployment"]
        Config["ManagedProxyConfiguration<br/>update spec"]
        ProxyServer["proxy-server<br/>update Deployment"]
        Hash["proxyAgent config<br/>update spec hash"]
        Rollout["OCM add-on manager<br/>ProgressivePerGroup"]
        Groups["progress through decision groups<br/>dev → stg → prod<br/>success + minSuccessTime before next group"]
        AddOn["ManagedClusterAddOn in current group<br/>Configured=True"]
        Render["cluster-proxy manager<br/>render agent chart"]
        Work["update ManifestWork"]
    end

    subgraph Spoke["spoke clusters in the current group"]
        direction TB
        WorkAgent["work-agent"]
        ProxyAgent["proxy-agent<br/>update Deployment"]
    end

    Upgrade -->|Helm updates directly| Manager
    Upgrade -->|Helm updates directly| Config
    Config -->|proxyServer.image<br/>not part of rollout| ProxyServer
    Config -->|proxyAgent.image<br/>part of rollout| Hash
    Hash --> Rollout
    Rollout --> Groups
    Groups -->|current group only| AddOn
    AddOn --> Render
    Render --> Work
    Work --> WorkAgent
    WorkAgent --> ProxyAgent
    WorkAgent -.->|Applied / Available| Work
    Work -.->|hash matches + Ready| Rollout

    classDef immediate fill:#fff3cd,stroke:#a66b00,color:#332200;
    classDef staged fill:#e8f3ff,stroke:#2563a6,color:#102a43;
    classDef spoke fill:#eaf7ed,stroke:#2f855a,color:#173d2a;
    class Manager,Config,ProxyServer immediate;
    class Hash,Rollout,Groups,AddOn,Render,Work staged;
    class WorkAgent,ProxyAgent spoke;

Yellow indicates updates that happen immediately on the Hub. Blue indicates updates controlled by ProgressivePerGroup, and green indicates processing on the spoke clusters. Dashed lines represent status reported from the spokes back to the Hub.

Configuration

1. Add environment labels to ManagedClusters

Split the ManagedCluster resources into dev, stg, and prod environments.

apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  name: dev-cluster
  labels:
    cluster-proxy: enabled
    environment: dev
spec:
  hubAcceptsClient: true
  leaseDurationSeconds: 60
---
apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  name: stg-cluster
  labels:
    cluster-proxy: enabled
    environment: stg
spec:
  hubAcceptsClient: true
  leaseDurationSeconds: 60
---
apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  name: prod-cluster
  labels:
    cluster-proxy: enabled
    environment: prod
spec:
  hubAcceptsClient: true
  leaseDurationSeconds: 60

2. Define the rollout order in the Placement

The Placement selects only clusters that have cluster-proxy: enabled and whose environment is dev, stg, or prod. The selected clusters are then divided into decision groups by environment.

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: cluster-proxy-placement
  namespace: open-cluster-management
spec:
  predicates:
  - requiredClusterSelector:
      labelSelector:
        matchLabels:
          cluster-proxy: enabled
        matchExpressions:
        - key: environment
          operator: In
          values:
          - dev
          - stg
          - prod
  decisionStrategy:
    groupStrategy:
      clustersPerDecisionGroup: "100%"
      decisionGroups:
      - groupName: dev
        groupClusterSelector:
          labelSelector:
            matchLabels:
              environment: dev
      - groupName: stg
        groupClusterSelector:
          labelSelector:
            matchLabels:
              environment: stg
      - groupName: prod
        groupClusterSelector:
          labelSelector:
            matchLabels:
              environment: prod

The rollout order is determined by the order of entries under decisionGroups. The Placement controller assigns group index 0 to dev, 1 to stg, and 2 to prod, and ProgressivePerGroup distributes the configuration in that order.

3. Configure the rollout on ClusterManagementAddOn

Configure ProgressivePerGroup on the cluster-proxy ClusterManagementAddOn.

apiVersion: addon.open-cluster-management.io/v1beta1
kind: ClusterManagementAddOn
metadata:
  name: cluster-proxy
  annotations:
    addon.open-cluster-management.io/lifecycle: addon-manager
spec:
  addOnMeta:
    displayName: cluster-proxy
    description: cluster-proxy
  defaultConfigs:
  - group: proxy.open-cluster-management.io
    resource: managedproxyconfigurations
    name: cluster-proxy
  installStrategy:
    type: Placements
    placements:
    - name: cluster-proxy-placement
      namespace: open-cluster-management
      rolloutStrategy:
        type: ProgressivePerGroup
        progressivePerGroup:
          minSuccessTime: 5m
          progressDeadline: 15m
          maxFailures: 0

4. Update the add-on configuration

Update the ManagedProxyConfiguration referenced by spec.defaultConfigs. When you change the tag in the standard cluster-proxy Helm chart, the proxyServer.image and proxyAgent.image fields in this resource are also updated. Replace vX.Y.Z with the version you want to roll out.

apiVersion: proxy.open-cluster-management.io/v1alpha1
kind: ManagedProxyConfiguration
metadata:
  name: cluster-proxy
spec:
  authentication:
    dump:
      secrets: {}
    signer:
      type: SelfSigned
  proxyServer:
    image: quay.io/open-cluster-management/cluster-proxy:vX.Y.Z
    replicas: 1
    namespace: open-cluster-management
    entrypoint:
      type: PortForward
    port: 8091
  proxyAgent:
    image: quay.io/open-cluster-management/cluster-proxy:vX.Y.Z
    replicas: 1

Verify the rollout

When the configuration spec changes, a rollout starts using the new spec hash. The expected sequence is:

Apply the new configuration to dev.
Once dev reaches a successful state and remains successful for the minSuccessTime value of five minutes, proceed to stg.
Perform the same check for stg, then proceed to prod.

You can inspect the groups created by the Placement with the following commands:

kubectl -n open-cluster-management get placement cluster-proxy-placement -o yaml

kubectl -n open-cluster-management get placementdecision \
  -l cluster.open-cluster-management.io/placement=cluster-proxy-placement \
  -L cluster.open-cluster-management.io/decision-group-index \
  -L cluster.open-cluster-management.io/decision-group-name

Verify that dev, stg, and prod all exist and that their group indexes are 0, 1, and 2 in that order. The rollout status of each cluster is reflected in ManagedClusterAddOn, while the status of the deployed manifests is reflected in ManifestWork.

kubectl get managedclusteraddon -A \
  --field-selector metadata.name=cluster-proxy
kubectl get manifestwork -A -l open-cluster-management.io/addon-name=cluster-proxy

What happens when a rollout fails

A cluster that does not reach a successful state within the 15 minutes configured by progressDeadline is treated as timed out. With maxFailures: 0, the rollout does not proceed to the next group while any cluster is failed or timed out. If the affected cluster recovers later, the rollout automatically resumes after the cluster remains successful for the configured minSuccessTime.

Caveats

Hub-side updates are not staged

Both cluster-proxy-addon-manager, which is managed directly by Helm, and cluster-proxy-addon-user, when service proxy is enabled, are updated immediately on the Hub. The proxy-server created from ManagedProxyConfiguration.spec.proxyServer.image also does not wait for the per-cluster rollout.

As a result, during the rollout, the Hub may be running the new version of proxy-server while the proxy-agents in stg and prod are still running the old version. If the old and new versions are not compatible, clusters whose agents have not yet been updated can also be affected.

If only the agent chart embedded in the new manager image or the rendering logic changes, while the referenced configuration spec remains unchanged, the hash does not change. In that case, no rollout starts, and multiple ManifestWork resources may be updated at once. In other words, what can be validated progressively here is a proxy-agent configuration change, not the entire cluster-proxy release.

Unclassified clusters and empty groups

Clusters that match requiredClusterSelector but do not match any groupClusterSelector are not excluded. They are placed into an unnamed decision group and updated after the named groups. In this example, environment is restricted to dev, stg, and prod to prevent such clusters from being included accidentally.

Also, a group is not created when zero clusters match it. For example, if there are no dev clusters, stg becomes group index 0 and prod becomes group index 1. Be sure to inspect the actual decision groups before starting the rollout.

Example of generated resources

If there is one cluster in each environment, the selection result looks like this:

group index	group	PlacementDecision	cluster
0	dev	cluster-proxy-placement-decision-1	dev-cluster
1	stg	cluster-proxy-placement-decision-2	stg-cluster
2	prod	cluster-proxy-placement-decision-3	prod-cluster

The Placement updated by the Placement controller looks like this:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: cluster-proxy-placement
  namespace: open-cluster-management
spec:
  predicates:
  - requiredClusterSelector:
      labelSelector:
        matchLabels:
          cluster-proxy: enabled
        matchExpressions:
        - key: environment
          operator: In
          values:
          - dev
          - stg
          - prod
  decisionStrategy:
    groupStrategy:
      clustersPerDecisionGroup: "100%"
      decisionGroups:
      - groupName: dev
        groupClusterSelector:
          labelSelector:
            matchLabels:
              environment: dev
      - groupName: stg
        groupClusterSelector:
          labelSelector:
            matchLabels:
              environment: stg
      - groupName: prod
        groupClusterSelector:
          labelSelector:
            matchLabels:
              environment: prod
status:
  numberOfSelectedClusters: 3
  decisionGroups:
  - decisionGroupIndex: 0
    decisionGroupName: dev
    decisions:
    - cluster-proxy-placement-decision-1
    clusterCount: 1
  - decisionGroupIndex: 1
    decisionGroupName: stg
    decisions:
    - cluster-proxy-placement-decision-2
    clusterCount: 1
  - decisionGroupIndex: 2
    decisionGroupName: prod
    decisions:
    - cluster-proxy-placement-decision-3
    clusterCount: 1
  conditions:
  - type: PlacementMisconfigured
    status: "False"
    reason: Succeedconfigured
    message: Placement configurations check pass
    lastTransitionTime: "<timestamp>"
  - type: PlacementSatisfied
    status: "True"
    reason: AllDecisionsScheduled
    message: All cluster decisions scheduled
    lastTransitionTime: "<timestamp>"

From this Placement, the following PlacementDecision resources are generated for each environment:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: PlacementDecision
metadata:
  name: cluster-proxy-placement-decision-1
  namespace: open-cluster-management
  labels:
    cluster.open-cluster-management.io/placement: cluster-proxy-placement
    cluster.open-cluster-management.io/decision-group-index: "0"
    cluster.open-cluster-management.io/decision-group-name: dev
  ownerReferences:
  - apiVersion: cluster.open-cluster-management.io/v1beta1
    kind: Placement
    name: cluster-proxy-placement
    uid: "<Placement UID>"
    controller: true
    blockOwnerDeletion: true
status:
  decisions:
  - clusterName: dev-cluster
    reason: ""
---
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: PlacementDecision
metadata:
  name: cluster-proxy-placement-decision-2
  namespace: open-cluster-management
  labels:
    cluster.open-cluster-management.io/placement: cluster-proxy-placement
    cluster.open-cluster-management.io/decision-group-index: "1"
    cluster.open-cluster-management.io/decision-group-name: stg
  ownerReferences:
  - apiVersion: cluster.open-cluster-management.io/v1beta1
    kind: Placement
    name: cluster-proxy-placement
    uid: "<Placement UID>"
    controller: true
    blockOwnerDeletion: true
status:
  decisions:
  - clusterName: stg-cluster
    reason: ""
---
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: PlacementDecision
metadata:
  name: cluster-proxy-placement-decision-3
  namespace: open-cluster-management
  labels:
    cluster.open-cluster-management.io/placement: cluster-proxy-placement
    cluster.open-cluster-management.io/decision-group-index: "2"
    cluster.open-cluster-management.io/decision-group-name: prod
  ownerReferences:
  - apiVersion: cluster.open-cluster-management.io/v1beta1
    kind: Placement
    name: cluster-proxy-placement
    uid: "<Placement UID>"
    controller: true
    blockOwnerDeletion: true
status:
  decisions:
  - clusterName: prod-cluster
    reason: ""

The numeric suffix in a PlacementDecision name starts at 1, while the group index starts at 0. When checking the rollout order, look at the cluster.open-cluster-management.io/decision-group-index label rather than the resource name.

DEV Community