DEV Community

Kahiro Okina
Kahiro Okina

Posted on

Running Istio Ambient Multicluster with kind

#kubernetes #istio #servicemesh #network

Architecture

  • Create cluster1 / cluster2 using kind
  • Enable LoadBalancer support with cloud-provider-kind
  • Configure Istio Ambient mode in multi-primary / multi-network setup
  • Verify cross-cluster load balancing with sample applications

About This Guide

  • Using Helm instead of istioctl: Designed for integration with GitOps tools like ArgoCD/FluxCD
  • Configuration details: For detailed information about Istio Ambient mode multicluster configuration, refer to the official blog
  • Working directory: Execute all commands from the root directory of the Istio repository

Prerequisites

  • macOS + Docker Desktop
  • kubectl / kind / helm / cloud-provider-kind installed
  • Clone the Istio repository and work from its root directory: 
  git clone https://github.com/istio/istio.git
  cd istio
  # Execute all subsequent commands from this directory
Enter fullscreen mode Exit fullscreen mode

Steps

1. Create Two kind Clusters 

kind create cluster --name cluster1
kind create cluster --name cluster2
Enter fullscreen mode Exit fullscreen mode

2. Install Gateway API CRDs 

GATEWAY_API_VER=v1.4.1

kubectl apply --context kind-cluster1 --server-side --force-conflicts -f \
  "https://github.com/kubernetes-sigs/gateway-api/releases/download/${GATEWAY_API_VER}/experimental-install.yaml"

kubectl apply --context kind-cluster2 --server-side --force-conflicts -f \
  "https://github.com/kubernetes-sigs/gateway-api/releases/download/${GATEWAY_API_VER}/experimental-install.yaml"
Enter fullscreen mode Exit fullscreen mode

3. Start cloud-provider-kind

Launch in a separate terminal:

sudo cloud-provider-kind
Enter fullscreen mode Exit fullscreen mode

4. Configure the Same CA for Both Clusters

For mTLS to work in a multicluster setup, we need to use the same root CA:

kubectl create ns istio-system --context kind-cluster1
kubectl create ns istio-system --context kind-cluster2

# Use sample certificates from samples/certs/
kubectl create secret generic cacerts -n istio-system --context kind-cluster1 \
  --from-file=samples/certs/ca-cert.pem \
  --from-file=samples/certs/ca-key.pem \
  --from-file=samples/certs/root-cert.pem \
  --from-file=samples/certs/cert-chain.pem

kubectl create secret generic cacerts -n istio-system --context kind-cluster2 \
  --from-file=samples/certs/ca-cert.pem \
  --from-file=samples/certs/ca-key.pem \
  --from-file=samples/certs/root-cert.pem \
  --from-file=samples/certs/cert-chain.pem
Enter fullscreen mode Exit fullscreen mode

5. Install Istio

cluster1:

helm upgrade --install istio-base manifests/charts/base \
  -n istio-system --kube-context kind-cluster1

cat <<'EOF' | helm upgrade --install istiod manifests/charts/istio-control/istio-discovery \
  -n istio-system --kube-context kind-cluster1 \
  -f manifests/helm-profiles/ambient.yaml -f -
global:
  meshID: mesh1
  multiCluster:
    clusterName: cluster1
  network: network1
pilot:
  env:
    AMBIENT_ENABLE_MULTI_NETWORK: "true"
    ENABLE_WILDCARD_HOST_SERVICE_ENTRIES_FOR_TLS: "true"
EOF

cat <<'EOF' | helm upgrade --install istio-cni manifests/charts/istio-cni \
  -n istio-system --kube-context kind-cluster1 \
  -f manifests/helm-profiles/ambient.yaml -f -
global:
  meshID: mesh1
  multiCluster:
    clusterName: cluster1
  network: network1
EOF

cat <<'EOF' | helm upgrade --install ztunnel manifests/charts/ztunnel \
  -n istio-system --kube-context kind-cluster1 \
  -f manifests/helm-profiles/ambient.yaml -f -
global:
  meshID: mesh1
  multiCluster:
    clusterName: cluster1
  network: network1
EOF

kubectl label ns istio-system --context kind-cluster1 topology.istio.io/network=network1 --overwrite
Enter fullscreen mode Exit fullscreen mode

cluster2:

helm upgrade --install istio-base manifests/charts/base \
  -n istio-system --kube-context kind-cluster2

cat <<'EOF' | helm upgrade --install istiod manifests/charts/istio-control/istio-discovery \
  -n istio-system --kube-context kind-cluster2 \
  -f manifests/helm-profiles/ambient.yaml -f -
global:
  meshID: mesh1
  multiCluster:
    clusterName: cluster2
  network: network2
pilot:
  env:
    AMBIENT_ENABLE_MULTI_NETWORK: "true"
    ENABLE_WILDCARD_HOST_SERVICE_ENTRIES_FOR_TLS: "true"
EOF

cat <<'EOF' | helm upgrade --install istio-cni manifests/charts/istio-cni \
  -n istio-system --kube-context kind-cluster2 \
  -f manifests/helm-profiles/ambient.yaml -f -
global:
  meshID: mesh1
  multiCluster:
    clusterName: cluster2
  network: network2
EOF

cat <<'EOF' | helm upgrade --install ztunnel manifests/charts/ztunnel \
  -n istio-system --kube-context kind-cluster2 \
  -f manifests/helm-profiles/ambient.yaml -f -
global:
  meshID: mesh1
  multiCluster:
    clusterName: cluster2
  network: network2
EOF

kubectl label ns istio-system --context kind-cluster2 topology.istio.io/network=network2 --overwrite
Enter fullscreen mode Exit fullscreen mode

6. Exchange Remote Secrets 

# Handle differences between macOS and Linux base64 commands
b64dec() {
  if base64 --help 2>&1 | grep -q -- '--decode'; then
    base64 --decode
  else
    base64 -D
  fi
}

# Get IP addresses of kind control-plane nodes
SERVER_CLUSTER1="https://$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' cluster1-control-plane):6443"
SERVER_CLUSTER2="https://$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' cluster2-control-plane):6443"

# Create service account token for cluster1
kubectl apply --context kind-cluster1 -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: istio-reader-service-account-istio-remote-secret-token
  namespace: istio-system
  annotations:
    kubernetes.io/service-account.name: istio-reader-service-account
type: kubernetes.io/service-account-token
EOF

# Wait for token generation
until kubectl get --context kind-cluster1 -n istio-system secret istio-reader-service-account-istio-remote-secret-token \
  -o jsonpath='{.data.token}' 2>/dev/null | grep -q .; do
  sleep 1
done

TOKEN_CLUSTER1="$(kubectl get --context kind-cluster1 -n istio-system secret istio-reader-service-account-istio-remote-secret-token \
  -o jsonpath='{.data.token}' | b64dec)"
CA_B64_CLUSTER1="$(kubectl get --context kind-cluster1 -n istio-system secret istio-reader-service-account-istio-remote-secret-token \
  -o jsonpath='{.data.ca\.crt}')"

# Create secret in cluster2 to access cluster1
kubectl apply --context kind-cluster2 -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: istio-remote-secret-cluster1
  namespace: istio-system
  labels:
    istio/multiCluster: "true"
  annotations:
    networking.istio.io/cluster: cluster1
stringData:
  cluster1: |
    apiVersion: v1
    kind: Config
    clusters:
    - name: cluster1
      cluster:
        server: ${SERVER_CLUSTER1}
        certificate-authority-data: ${CA_B64_CLUSTER1}
    contexts:
    - name: cluster1
      context:
        cluster: cluster1
        user: cluster1
    current-context: cluster1
    users:
    - name: cluster1
      user:
        token: ${TOKEN_CLUSTER1}
EOF

# Create service account token for cluster2
kubectl apply --context kind-cluster2 -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: istio-reader-service-account-istio-remote-secret-token
  namespace: istio-system
  annotations:
    kubernetes.io/service-account.name: istio-reader-service-account
type: kubernetes.io/service-account-token
EOF

# Wait for token generation
until kubectl get --context kind-cluster2 -n istio-system secret istio-reader-service-account-istio-remote-secret-token \
  -o jsonpath='{.data.token}' 2>/dev/null | grep -q .; do
  sleep 1
done

TOKEN_CLUSTER2="$(kubectl get --context kind-cluster2 -n istio-system secret istio-reader-service-account-istio-remote-secret-token \
  -o jsonpath='{.data.token}' | b64dec)"
CA_B64_CLUSTER2="$(kubectl get --context kind-cluster2 -n istio-system secret istio-reader-service-account-istio-remote-secret-token \
  -o jsonpath='{.data.ca\.crt}')"

# Create secret in cluster1 to access cluster2
kubectl apply --context kind-cluster1 --server-side -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: istio-remote-secret-cluster2
  namespace: istio-system
  labels:
    istio/multiCluster: "true"
  annotations:
    networking.istio.io/cluster: cluster2
stringData:
  cluster2: |
    apiVersion: v1
    kind: Config
    clusters:
    - name: cluster2
      cluster:
        server: ${SERVER_CLUSTER2}
        certificate-authority-data: ${CA_B64_CLUSTER2}
    contexts:
    - name: cluster2
      context:
        cluster: cluster2
        user: cluster2
    current-context: cluster2
    users:
    - name: cluster2
      user:
        token: ${TOKEN_CLUSTER2}
EOF
Enter fullscreen mode Exit fullscreen mode

7. Create East-West Gateways 

kubectl apply --context kind-cluster1 --server-side -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: istio-eastwestgateway
  namespace: istio-system
  labels:
    topology.istio.io/network: "network1"
spec:
  gatewayClassName: "istio-east-west"
  listeners:
    - name: mesh
      port: 15008
      protocol: HBONE
      tls:
        mode: Terminate
        options:
          gateway.istio.io/tls-terminate-mode: ISTIO_MUTUAL
EOF

kubectl apply --context kind-cluster2 --server-side -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: istio-eastwestgateway
  namespace: istio-system
  labels:
    topology.istio.io/network: "network2"
spec:
  gatewayClassName: "istio-east-west"
  listeners:
    - name: mesh
      port: 15008
      protocol: HBONE
      tls:
        mode: Terminate
        options:
          gateway.istio.io/tls-terminate-mode: ISTIO_MUTUAL
EOF
Enter fullscreen mode Exit fullscreen mode

Wait for EXTERNAL-IP assignment:

for c in kind-cluster1 kind-cluster2; do
  echo "=== waiting EXTERNAL-IP for $c"
  until kubectl --context "$c" -n istio-system get svc istio-eastwestgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null | grep -Eq '[0-9]'; do
    sleep 2
  done
  kubectl --context "$c" -n istio-system get svc istio-eastwestgateway
done
Enter fullscreen mode Exit fullscreen mode

8. Verification

Create sample namespace and enable ambient mode:

kubectl create ns sample --context kind-cluster1
kubectl create ns sample --context kind-cluster2

kubectl label ns sample --context kind-cluster1 istio.io/dataplane-mode=ambient --overwrite
kubectl label ns sample --context kind-cluster2 istio.io/dataplane-mode=ambient --overwrite
Enter fullscreen mode Exit fullscreen mode

Deploy helloworld (v1 to cluster1, v2 to cluster2):

# Create Service in both clusters
kubectl apply --context kind-cluster1 -n sample -f samples/helloworld/helloworld.yaml -l service=helloworld
kubectl apply --context kind-cluster2 -n sample -f samples/helloworld/helloworld.yaml -l service=helloworld

# Create Deployment with different versions in each cluster
kubectl apply --context kind-cluster1 -n sample -f samples/helloworld/helloworld.yaml -l version=v1
kubectl apply --context kind-cluster2 -n sample -f samples/helloworld/helloworld.yaml -l version=v2

# Mark as global service (enable cross-cluster communication)
kubectl label svc helloworld --context kind-cluster1 -n sample istio.io/global=true --overwrite
kubectl label svc helloworld --context kind-cluster2 -n sample istio.io/global=true --overwrite
Enter fullscreen mode Exit fullscreen mode

Deploy curl:

kubectl apply --context kind-cluster1 -n sample -f samples/curl/curl.yaml
kubectl apply --context kind-cluster2 -n sample -f samples/curl/curl.yaml
Enter fullscreen mode Exit fullscreen mode

Verify cross-cluster load balancing:

# Call helloworld from curl in cluster1
for i in {1..10}; do
  kubectl exec --context kind-cluster1 -n sample deploy/curl -c curl -- \
    curl -sS "helloworld.sample:5000/hello"
done

# Also verify from cluster2
for i in {1..10}; do
  kubectl exec --context kind-cluster2 -n sample deploy/curl -c curl -- \
    curl -sS "helloworld.sample:5000/hello"
done
Enter fullscreen mode Exit fullscreen mode

Success if you see a mix of v1 and v2 responses.

Example:

$ for i in {1..10}; do
  kubectl exec --context kind-cluster1 -n sample deploy/curl -c curl -- \
    curl -sS "helloworld.sample:5000/hello"
done
Hello version: v1, instance: helloworld-v1-696f8879d6-w6g89
Hello version: v2, instance: helloworld-v2-59fc9f4558-g8r8b
Hello version: v1, instance: helloworld-v1-696f8879d6-w6g89
Hello version: v2, instance: helloworld-v2-59fc9f4558-g8r8b
Hello version: v1, instance: helloworld-v1-696f8879d6-w6g89
Hello version: v2, instance: helloworld-v2-59fc9f4558-g8r8b
Hello version: v1, instance: helloworld-v1-696f8879d6-w6g89
Hello version: v2, instance: helloworld-v2-59fc9f4558-g8r8b
Hello version: v2, instance: helloworld-v2-59fc9f4558-g8r8b
Hello version: v2, instance: helloworld-v2-59fc9f4558-g8r8b
Enter fullscreen mode Exit fullscreen mode

Common Pitfalls

  • cloud-provider-kind not running: EXTERNAL-IP remains <pending> and never gets assigned
  • CA mismatch: Errors like curl: (56) Recv failure: Connection reset by peer occur
  • Missing istio.io/global=true label: Cross-cluster communication won't happen unless applied to Services in both clusters
  • Running outside the Istio repository: Errors occur when samples/ or manifests/ directories are not found

References

Top comments (0)