In this article, I'll introduce you to HelmChartProxy, a powerful feature of Cluster API that enables bulk deployment of Helm charts across multiple Kubernetes clusters.
Introduction
When managing multiple Kubernetes clusters, you often need to deploy the same Helm charts across different environments. Traditionally, this meant running Helm commands individually for each cluster. However, with Cluster API's HelmChartProxy, you can now streamline this process with bulk deployments.
We'll explore:
- Bulk deployment to all clusters
- Targeted deployment using labels
- Deployment from private registries
Setting Up the Test Environment
Prerequisites
- Docker (16GB+ memory recommended)
- kubectl
- kind
- clusterctl
1. Creating the Management Cluster
First, let's create a management cluster using kind. Create the following configuration file:
# kind-cluster-with-extramounts.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
ipFamily: dual
nodes:
- role: control-plane
extraMounts:
- hostPath: /var/run/docker.sock
containerPath: /var/run/docker.sock
Create the cluster using this configuration:
kind create cluster --config kind-cluster-with-extramounts.yaml
2. Setting Up Cluster API
Install Cluster API on the management cluster:
export CLUSTER_TOPOLOGY=true
clusterctl init --infrastructure docker --addon helm
Wait for all components to be ready:
kubectl wait --for=condition=Available --timeout=300s -n capi-system deployments --all && \
kubectl wait --for=condition=Available --timeout=300s -n capi-kubeadm-bootstrap-system deployments --all && \
kubectl wait --for=condition=Available --timeout=300s -n capi-kubeadm-control-plane-system deployments --all && \
kubectl wait --for=condition=Available --timeout=300s -n capd-system deployments --all && \
kubectl wait --for=condition=Available --timeout=300s -n caaph-system deployments --all
3. Creating Workload Clusters
We'll create two clusters named "muscat" and "delaware":
# Generate manifest for muscat cluster
clusterctl generate cluster muscat \
--flavor development \
--kubernetes-version v1.28.0 \
--control-plane-machine-count=3 \
--worker-machine-count=3 \
> muscat.yaml
# Generate manifest for delaware cluster
clusterctl generate cluster delaware \
--flavor development \
--kubernetes-version v1.28.0 \
--control-plane-machine-count=3 \
--worker-machine-count=3 \
> delaware.yaml
# Create the clusters
kubectl apply -f muscat.yaml
kubectl apply -f delaware.yaml
Get the kubeconfig files:
clusterctl get kubeconfig muscat > muscat-kubeconfig.yaml
clusterctl get kubeconfig delaware > delaware-kubeconfig.yaml
4. Setting Up CNI
Install Calico on both clusters:
# Install Calico on muscat
kubectl --kubeconfig=muscat-kubeconfig.yaml apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml
# Install Calico on delaware
kubectl --kubeconfig=delaware-kubeconfig.yaml apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml
Deploying with HelmChartProxy
Deploying to All Clusters
Let's look at an example of deploying nginx to all clusters:
# helm-chart-proxy-nginx.yaml
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
name: nginx
spec:
clusterSelector: {} # Empty selector means "select all clusters"
repoURL: oci://registry-1.docker.io/bitnamicharts
chartName: nginx
version: "18.2.5"
releaseName: nginx
namespace: nginx
options:
waitForJobs: true
atomic: true
wait: true
timeout: 5m
install:
createNamespace: true
valuesTemplate: |-
service:
type: NodePort
Deploy and verify:
# Deploy nginx
kubectl apply -f helm-chart-proxy-nginx.yaml
# Wait for HelmReleaseProxy to be ready
kubectl wait --for=condition=Ready --timeout=300s helmreleaseproxy --all
# Check pods on muscat
kubectl --kubeconfig=muscat-kubeconfig.yaml get pods -n nginx
# Check pods on delaware
kubectl --kubeconfig=delaware-kubeconfig.yaml get pods -n nginx
Targeted Deployment Using Labels
You can deploy to specific clusters using labels:
# helm-chart-proxy-nginx.yaml
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
name: nginx
spec:
clusterSelector:
matchLabels:
use-nginx: "true" # Only deploy to clusters with this label
# ... (other settings remain the same)
Usage:
# Deploy nginx
kubectl apply -f helm-chart-proxy-nginx.yaml
# Label the muscat cluster
kubectl label cluster muscat use-nginx=true
# Check deployment status
kubectl get helmreleaseproxy -A
# Verify nginx is running on muscat
kubectl --kubeconfig=muscat-kubeconfig.yaml get pods -n nginx
# Verify nginx is not running on delaware
kubectl --kubeconfig=delaware-kubeconfig.yaml get pods -n nginx
Deploying from Private Registries
Let's use GitHub Container Registry as an example:
Create a GitHub Personal Access Token (needs read:packages permission)
Set up registry credentials:
# Create config.json with base64 encoded credentials
echo -n "YOUR_GITHUB_USERNAME:YOUR_GITHUB_TOKEN" | base64 > auth.txt
cat > config.json << EOF
{
"auths": {
"ghcr.io": {
"auth": "$(cat auth.txt)"
}
}
}
EOF
# Create the secret
kubectl create secret generic github-creds \
--from-file=config.json \
-n caaph-system
- Create HelmChartProxy for private charts:
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
name: private-chart
spec:
clusterSelector: {}
repoURL: oci://ghcr.io/YOUR_GITHUB_USERNAME
chartName: YOUR_CHART_NAME
version: "0.1.0"
credentials:
secret:
name: github-creds
namespace: caaph-system
key: config.json
# ... (other settings)
Conclusion
HelmChartProxy significantly simplifies the process of deploying Helm charts across multiple clusters. Key benefits include:
- Reduced operational overhead through bulk deployments
- Flexible deployment control using labels
- Support for private registries
If you're managing multiple Kubernetes clusters, consider incorporating HelmChartProxy into your workflow. It can greatly streamline your Helm chart deployment process and make your multi-cluster management more efficient.
Top comments (0)