DEV Community: Luiz Cesar Cherri

Managing access credentials is challenging for security and DevOps teams. In this article, I share how to use Service Accounts and RHACM/OCM to automate centralized multicluster access.

Luiz Cesar Cherri — Tue, 01 Jul 2025 10:50:38 +0000

Luiz Cesar Cherri

Jun 30 '25

Managing Service Accounts in Multicluster Environments with RHACM

#devops #security #kubernetes #openshift

Comments

7 min read

Managing Service Accounts in Multicluster Environments with RHACM

Luiz Cesar Cherri — Mon, 30 Jun 2025 17:59:50 +0000

Service Accounts are fundamental Kubernetes resources used to enable programmatic access to the cluster, typically for automation, workloads, or system integrations. They provide access tokens that authenticate against the Kubernetes API, offering a simpler alternative to traditional X.509 certificate-based authentication or OAuth mechanisms, such as those commonly used in OpenShift clusters.

Service Account - A Brief History

Since the early days of Kubernetes, administrators could create ServiceAccount tokens without an expiration date. While this approach was convenient, it introduced security risks - if a token was leaked, it could be used indefinitely until an administrator revoked it manually.

To address this, Kubernetes and OpenShift deprecated long-lived tokens. From now on, ServiceAccount tokens are time-bound, meaning they have an expiration date and must be renewed periodically.

This change was introduced starting with Kubernetes v1.24, when the automatic creation of long-lived legacy tokens was disabled by default. Instead, Kubernetes adopted the Bound ServiceAccount Token Volume feature, which issues short-lived, auto-rotating tokens tied to the pod lifecycle. This approach improves security but also introduces operational challenges, especially in scenarios where tokens are required outside the pod context - such as CI/CD pipelines, external automation, or multicluster management.

What is a Managed ServiceAccount?

Managing ServiceAccount tokens at scale — especially across multiple clusters — introduces operational challenges. Each interaction with a Kubernetes API requires a fresh, valid token. With tokens now being short-lived and requiring frequent rotation, DevOps teams need a reliable way to automate their issuance, renewal, and distribution.

A Managed ServiceAccount is a custom resource definition (CRD) introduced by Open Cluster Management. It lets you define a ServiceAccount that is automatically created on all eligible managed clusters, based on criteria defined by your DevOps team.

It ensures that the ServiceAccount is consistently created across clusters, and it automatically handles token issuance and rotation, making secure multicluster access much easier to manage.

Note: Managed ServiceAccounts focus on token lifecycle, but they do not manage RBAC permissions. Those remain the responsibility of cluster administrators, often addressed through GitOps practices or policies.

Red Hat Advanced Cluster Management for Kubernetes (RHACM) is the downstream distribution of the upstream Open Cluster Management (OCM) project.

Although this article uses Red Hat OpenShift and RHACM to demonstrate the examples, the features, APIs, and resource definitions are the same in both RHACM and OCM.

Less Talk, More Action

Once you have a functional RHACM or OCM environment — preferably composed of a Hub Cluster and one or more Managed Clusters — you can create a resource like the following on the Hub Cluster:

apiVersion: authentication.open-cluster-management.io/v1beta1
kind: ManagedServiceAccount
metadata:
  name: automation
  namespace: <cluster-name>
spec:
  rotation:
    enabled: true
    validity: 2h

When a cluster is imported (i.e., becomes a Managed Cluster) into the Hub, the Hub creates a dedicated namespace for that cluster. This namespace is used to host configurations and resources managed by the Hub for that specific cluster.

To create a ServiceAccount on a Managed Cluster, the resource definition must be placed within the corresponding namespace on the Hub. Therefore, make sure to replace cluster-name with the name of your target Managed Cluster.

The YAML above defines a ServiceAccount named automation that will be created on the target Managed Cluster. The token associated with this ServiceAccount is configured to be automatically rotated every two hours.

As shown in the diagram, this definition not only creates a ServiceAccount on the target clusters, but also creates a Secret in the cluster’s namespace on the Hub. This Secret contains the most up-to-date token for the corresponding ServiceAccount.

In short, any tool or automation that needs access to the managed clusters can simply retrieve the token from the corresponding Secret in the Hub cluster.

It’s important to note that the Hub maintains and automatically rotates this token according to the configuration defined in the ManagedServiceAccount resource. This ensures that the token is always valid and up to date, without requiring manual intervention.

What About ServiceAccount Permissions?

As mentioned earlier, Managed ServiceAccounts handle the creation and automatic renewal of the ServiceAccount and its token. However, once the ServiceAccount is created on the managed cluster, it initially has no permissions to perform any operations within that cluster.

To grant the necessary permissions, you must configure a RoleBinding or ClusterRoleBinding on the managed cluster. This binding associates the ServiceAccount with the appropriate RBAC roles, ensuring that the tool or automation can operate as expected.

The example below shows a ClusterRoleBinding created for this purpose:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: automation
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: automation
  namespace: open-cluster-management-agent-addon

Note: The ServiceAccount is always created in the open-cluster-management-agent-addon namespace on the managed cluster.

The example above grants the automation ServiceAccount cluster-admin privileges. If more restrictive permissions are needed, you can create a custom Role or ClusterRole to define exactly the level of access required.

Scaling the Solution with Policies

So far, we’ve described the basic configuration of Managed ServiceAccounts. However, we haven’t yet covered how to make this process scalable and repeatable across multiple clusters.

In other words, the key question is:
“How can I automate this entire process so that whenever a new cluster is attached to the Hub, it immediately becomes accessible to my automation tools?”

To achieve this, we can leverage the Policy and Placement resources provided by RHACM/OCM.

These resources define “what” should be applied (the desired configuration) and “where” it should be applied (the set of managed clusters selected by Placement criteria).

The diagram below illustrates how this mechanism works:

This model allows you to automatically distribute or configure any resource — including Managed ServiceAccounts, RoleBindings, and others — based on the labels assigned to clusters. As soon as a managed cluster joins the Hub, the selected policies are applied automatically.

You can learn more about Policies and Placements in the Red Hat Official Documentation.

Putting It All Together

To support the technical concepts covered in this article, I’ve created a Git repository with Policy definitions that configure both the Managed ServiceAccount and the corresponding ClusterRoleBindings across all clusters imported into a given Hub Cluster.

lccherri / rhacm-managed-serviceaccount

Managing Service Accounts with RHACM

Managed Service Accounts in Red Hat Advanced Cluster Management (RHACM) allow you to securely distribute Service Accounts across managed clusters. The associated tokens are stored in dedicated secrets on the hub cluster, ensuring centralized and secure access.

This setup enables automation workflows to retrieve Service Account tokens directly from the hub cluster, simplifying and securing access to managed clusters.

This demonstration provides RHACM Policies that create all necessary resources to deliver this capability in a highly scalable environment.

Prerequisites

Before you begin, ensure the following requirements are met:

OpenShift Container Platform version 4.16 or later.
Red Hat Advanced Cluster Management for Kubernetes (RHACM) version 2.13 or later.

Installing RHACM

To install RHACM, execute the following command:

# Run the ACM installation script
sh ./00-rhacm/acm-install.sh

This script installs RHACM operator on your cluster.

Access the OpenShift console and navigate to Installed Operators > Advanced Cluster Management…

View on GitHub

With the approach implemented in this repository, whenever a new cluster is imported into the Hub, it immediately becomes accessible using a predefined ServiceAccount.

The example defines two Policies:

Managed ServiceAccount Policy:
- Creates the Managed ServiceAccount resource in the Hub.
- This Policy targets every namespace that has the label cluster.open-cluster-management.io/managedCluster.
ClusterRoleBinding Policy
- Applies a ClusterRoleBinding on the managed clusters.
- Grants the created ServiceAccount the necessary permissions to operate on those clusters.

Note: A Policy can also be used to define more restrictive Roles or ClusterRoles, depending on the level of access required.

The Big Picture

Now that you understand the concepts and how to implement them, let’s look at how this works in a real-world automation scenario.

The diagram below represents an Ansible automation workflow, where the tokens required to authenticate against the managed clusters are requested from the Hub Cluster:

The automation tool queries the Hub Cluster for one or more tokens to authenticate with the target managed clusters.
→ Tip: You can create a dedicated ServiceAccount on the Hub Cluster with restricted permissions that allow it to query only the Secrets related to specific Managed ServiceAccounts.
→ This approach helps enforce the principle of least privilege, improving overall security.
The Hub Cluster API returns the required tokens, extracted from the Secrets maintained by the Managed ServiceAccount controller.
The automation tool uses the retrieved tokens to perform API calls directly against the Managed Clusters, executing its intended tasks (deployments, queries, validations, etc.).
Meanwhile, the Hub Cluster and Managed Clusters continuously ensure that all ServiceAccount tokens remain fresh and valid, thanks to the automatic rotation and renewal mechanisms provided by Managed ServiceAccounts.

Note: If your use case requires short-lived tokens (less than two hours) or involves long-running automation workflows, it’s recommended that the automation tool retrieves a fresh token from the Hub before each interaction with the Managed Clusters.

This helps prevent authentication failures caused by token expiration during long-running workflows.

Final Thoughts

Managing ServiceAccounts at scale has always been a challenge in Kubernetes environments — even more so when operating across multiple clusters.

The combination of Managed ServiceAccounts and Policies in RHACM/OCM provides a robust and secure framework to automate identity management for workloads and automation tools.

This approach centralizes token management, ensures automatic rotation, and eliminates the need for manual credential handling on each cluster. As a result, it not only improves operational efficiency but also significantly enhances security posture.

Whether you’re operating CI/CD pipelines, external automation tools like Ansible, or any system that needs API-level access to clusters, this model provides a scalable, repeatable, and secure way to do it.

References

Highly Scalable Infrastructures with GitOps Pull Model

Luiz Cesar Cherri — Wed, 18 Jun 2025 12:38:49 +0000

In recent years, working as a consultant in the technology field, I’ve observed how companies have increasingly adopted multi-cluster strategies using Kubernetes and OpenShift. These strategies generally aim to distribute workloads across clusters for reasons such as team segmentation or to enhance system resilience and availability.

To support this approach, delivery automation and pipeline standardization have become essential to ensure consistency, auditability, and scalability — while also providing operational control and efficiency.

In this article, I show how Red Hat Advanced Cluster Management for Kubernetes (RHACM) enables a Hub-and-Spoke GitOps architecture integrated with Argo CD.

The practical foundation for this article is the GitOps Pull Model repository, which provides a simplified implementation of the proposed model.

But… What is GitOps?

GitOps is a declarative approach to infrastructure and application management that relies on Git as the single source of truth and leverages continuous automation to apply changes. Instead of executing manual commands or scripts directly against clusters, the desired state of the environment is versioned in a Git repository, and automated tools, like Argo CD, are responsible for reconciling that state with the actual environment.

This strategy brings several key benefits:

Auditability and traceability: Every change is recorded as a Git commit, enabling full visibility and review history.
Reproducibility and consistency: Environments can be rebuilt from the repository, minimizing drift and human error.
Security and control: By separating change approval (via Git) from change application (via automation), access to clusters can be tightly controlled.
Speed and scalability: Applications and configurations can be propagated efficiently and safely across multiple clusters.

GitOps extends familiar software development concepts — such as pull requests, CI/CD, and versioning — into the realm of infrastructure operations, unifying delivery processes with modern DevOps practices.

What is a Hub-and-Spoke Architecture?

In environments with only a few Kubernetes clusters, it’s common for each cluster to have its own Argo CD instance, directly managed by the development team or even a dedicated DevOps team. However, this approach becomes difficult to scale in dynamic environments where clusters are frequently provisioned and decommissioned.

To address this challenge, the Hub-and-Spoke model offers a well-established architectural pattern in distributed systems, where a central component (the Hub) coordinates or manages multiple peripheral components (the Spokes). In Kubernetes or OpenShift environments, this pattern proves especially useful for centrally managing a large number of clusters.

In a GitOps and multi-cluster management context, the architecture can be described as follows:

Hub: A central cluster, often running management tools like Red Hat Advanced Cluster Management (RHACM), responsible for registering, inspecting, and applying policies to the Spoke clusters.
Spokes: Managed clusters that operate independently but are governed by configurations or policies originating from the Hub. Each Spoke typically runs its own GitOps agent (e.g., Argo CD) and pulls its desired state from a shared Git repository.

The RHACM GitOps - Pull Model

Implementing a Hub-and-Spoke architecture can require significant effort from DevOps teams. To simplify this process, Red Hat Advanced Cluster Management (RHACM) provides built-in mechanisms that streamline the deployment of this model.

RHACM includes controllers that process Argo CD ApplicationSets and distribute the resulting Applications to managed clusters that match predefined Placement criteria. Notably, the Hub cluster does not execute or reconcile these applications itself. Instead, it delegates this responsibility to the target managed clusters.

The diagram below illustrates how this delegation and synchronization flow works in practice:

Once the Argo CD Applications are applied to the managed clusters, they begin reporting their status back to the Hub cluster.

For more details on this capability, refer to the official documentation:
Red Hat Advanced Cluster Management — GitOps Integration Guide

Bridging Theory and Practice

The GitOps Pull Model repository presents a simplified implementation of the GitOps Hub-and-Spoke architecture, where each Spoke cluster operates independently and pulls from a shared Git repository. Although intentionally minimalistic, the project effectively demonstrates the core principles of GitOps in multi-cluster environments — with a key role played by Red Hat Advanced Cluster Management (RHACM).

The Role of RHACM

RHACM acts as the control plane for the Hub cluster, automating and orchestrating the following tasks:

Automated installation of the OpenShift GitOps Operator: RHACM applies policies that ensure all clusters have the necessary operator installed, enabling the provisioning of Argo CD instances.
Declarative Argo CD configuration on the Hub cluster: A dedicated policy configures the Argo CD instance on the Hub, centralizing control and initial setup.
Provisioning Argo CD on managed clusters: RHACM applies policies that deploy and configure Argo CD on each Spoke, so each cluster operates its own independent GitOps instance.
Selective artifact distribution via Placements: Through the use of Placements, RHACM informs its Argo CD instance which clusters must receive the configurations, based on labels assigned to the managed clusters. This allows centralized governance with flexible scope.

This architecture balances centralized governance and visibility (via the Hub) with decentralized execution (via the Spokes), enhancing security, scalability, and standardization across clusters.

Benefits of the Pull Model in Multi-Cluster Environments

Adopting a GitOps approach based on the pull model, especially within a Hub-and-Spoke architecture, unlocks several strategic and operational advantages. This choice directly promotes:

Enhanced Security
Communication flows from the managed clusters to the Git repository, removing the need for direct Hub access and significantly reducing the attack surface of the overall system.
Operational Autonomy for Spokes
Each managed cluster runs its own Argo CD instance and applies its own manifests, enabling independent operation with local control and greater agility in managing workloads.
Horizontal Scalability
Adding new clusters does not impact the Hub’s performance, as each Spoke synchronizes independently and asynchronously, allowing the architecture to scale linearly and efficiently.
Native Auditability and Traceability
With all configurations version-controlled in Git, changes can be tracked accurately, history is preserved, and rollbacks are safe and straightforward to execute.
Seamless Integration with DevSecOps Practices
Centralizing the desired state in Git supports automated validations, pull request-based code reviews, and CI/CD pipelines that are decoupled from runtime environments.
Elimination of Single Points of Failure
Since each Spoke manages its own sync operations, a failure in the Hub does not affect the Spokes, which continue operating based on the last known Git state — improving overall system resilience.

Conclusion

The GitOps Pull Model, provided by RHACM, offers a powerful foundation for scalable, secure, and autonomous Kubernetes and OpenShift operations. By adopting this approach, organizations can reduce operational complexity and improve governance across distributed environments.

Credits and References

This article was written to provide theoretical background for the GitOps Pull Model Repository:

GitOps Pull Model Sample

… which showcases a working example of a multi-cluster GitOps architecture using Red Hat Advanced Cluster Management (RHACM).

Both the conceptual and practical aspects are based on the following reference: