DEV Community: Kahiro Okina

Running Istio Ambient Multicluster with kind

Kahiro Okina — Mon, 19 Jan 2026 05:45:32 +0000

Architecture

Create cluster1 / cluster2 using kind
Enable LoadBalancer support with cloud-provider-kind
Configure Istio Ambient mode in multi-primary / multi-network setup
Verify cross-cluster load balancing with sample applications

About This Guide

Using Helm instead of istioctl: Designed for integration with GitOps tools like ArgoCD/FluxCD
Configuration details: For detailed information about Istio Ambient mode multicluster configuration, refer to the official blog
Working directory: Execute all commands from the root directory of the Istio repository

Prerequisites

macOS + Docker Desktop
kubectl / kind / helm / cloud-provider-kind installed
Clone the Istio repository and work from its root directory:

  git clone https://github.com/istio/istio.git
  cd istio
  # Execute all subsequent commands from this directory

Steps

1. Create Two kind Clusters

kind create cluster --name cluster1
kind create cluster --name cluster2

2. Install Gateway API CRDs

GATEWAY_API_VER=v1.4.1

kubectl apply --context kind-cluster1 --server-side --force-conflicts -f \
  "https://github.com/kubernetes-sigs/gateway-api/releases/download/${GATEWAY_API_VER}/experimental-install.yaml"

kubectl apply --context kind-cluster2 --server-side --force-conflicts -f \
  "https://github.com/kubernetes-sigs/gateway-api/releases/download/${GATEWAY_API_VER}/experimental-install.yaml"

3. Start cloud-provider-kind

Launch in a separate terminal:

sudo cloud-provider-kind

4. Configure the Same CA for Both Clusters

For mTLS to work in a multicluster setup, we need to use the same root CA:

kubectl create ns istio-system --context kind-cluster1
kubectl create ns istio-system --context kind-cluster2

# Use sample certificates from samples/certs/
kubectl create secret generic cacerts -n istio-system --context kind-cluster1 \
  --from-file=samples/certs/ca-cert.pem \
  --from-file=samples/certs/ca-key.pem \
  --from-file=samples/certs/root-cert.pem \
  --from-file=samples/certs/cert-chain.pem

kubectl create secret generic cacerts -n istio-system --context kind-cluster2 \
  --from-file=samples/certs/ca-cert.pem \
  --from-file=samples/certs/ca-key.pem \
  --from-file=samples/certs/root-cert.pem \
  --from-file=samples/certs/cert-chain.pem

5. Install Istio

cluster1:

helm upgrade --install istio-base manifests/charts/base \
  -n istio-system --kube-context kind-cluster1

cat <<'EOF' | helm upgrade --install istiod manifests/charts/istio-control/istio-discovery \
  -n istio-system --kube-context kind-cluster1 \
  -f manifests/helm-profiles/ambient.yaml -f -
global:
  meshID: mesh1
  multiCluster:
    clusterName: cluster1
  network: network1
pilot:
  env:
    AMBIENT_ENABLE_MULTI_NETWORK: "true"
    ENABLE_WILDCARD_HOST_SERVICE_ENTRIES_FOR_TLS: "true"
EOF

cat <<'EOF' | helm upgrade --install istio-cni manifests/charts/istio-cni \
  -n istio-system --kube-context kind-cluster1 \
  -f manifests/helm-profiles/ambient.yaml -f -
global:
  meshID: mesh1
  multiCluster:
    clusterName: cluster1
  network: network1
EOF

cat <<'EOF' | helm upgrade --install ztunnel manifests/charts/ztunnel \
  -n istio-system --kube-context kind-cluster1 \
  -f manifests/helm-profiles/ambient.yaml -f -
global:
  meshID: mesh1
  multiCluster:
    clusterName: cluster1
  network: network1
EOF

kubectl label ns istio-system --context kind-cluster1 topology.istio.io/network=network1 --overwrite

cluster2:

helm upgrade --install istio-base manifests/charts/base \
  -n istio-system --kube-context kind-cluster2

cat <<'EOF' | helm upgrade --install istiod manifests/charts/istio-control/istio-discovery \
  -n istio-system --kube-context kind-cluster2 \
  -f manifests/helm-profiles/ambient.yaml -f -
global:
  meshID: mesh1
  multiCluster:
    clusterName: cluster2
  network: network2
pilot:
  env:
    AMBIENT_ENABLE_MULTI_NETWORK: "true"
    ENABLE_WILDCARD_HOST_SERVICE_ENTRIES_FOR_TLS: "true"
EOF

cat <<'EOF' | helm upgrade --install istio-cni manifests/charts/istio-cni \
  -n istio-system --kube-context kind-cluster2 \
  -f manifests/helm-profiles/ambient.yaml -f -
global:
  meshID: mesh1
  multiCluster:
    clusterName: cluster2
  network: network2
EOF

cat <<'EOF' | helm upgrade --install ztunnel manifests/charts/ztunnel \
  -n istio-system --kube-context kind-cluster2 \
  -f manifests/helm-profiles/ambient.yaml -f -
global:
  meshID: mesh1
  multiCluster:
    clusterName: cluster2
  network: network2
EOF

kubectl label ns istio-system --context kind-cluster2 topology.istio.io/network=network2 --overwrite

6. Exchange Remote Secrets

# Handle differences between macOS and Linux base64 commands
b64dec() {
  if base64 --help 2>&1 | grep -q -- '--decode'; then
    base64 --decode
  else
    base64 -D
  fi
}

# Get IP addresses of kind control-plane nodes
SERVER_CLUSTER1="https://$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' cluster1-control-plane):6443"
SERVER_CLUSTER2="https://$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' cluster2-control-plane):6443"

# Create service account token for cluster1
kubectl apply --context kind-cluster1 -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: istio-reader-service-account-istio-remote-secret-token
  namespace: istio-system
  annotations:
    kubernetes.io/service-account.name: istio-reader-service-account
type: kubernetes.io/service-account-token
EOF

# Wait for token generation
until kubectl get --context kind-cluster1 -n istio-system secret istio-reader-service-account-istio-remote-secret-token \
  -o jsonpath='{.data.token}' 2>/dev/null | grep -q .; do
  sleep 1
done

TOKEN_CLUSTER1="$(kubectl get --context kind-cluster1 -n istio-system secret istio-reader-service-account-istio-remote-secret-token \
  -o jsonpath='{.data.token}' | b64dec)"
CA_B64_CLUSTER1="$(kubectl get --context kind-cluster1 -n istio-system secret istio-reader-service-account-istio-remote-secret-token \
  -o jsonpath='{.data.ca\.crt}')"

# Create secret in cluster2 to access cluster1
kubectl apply --context kind-cluster2 -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: istio-remote-secret-cluster1
  namespace: istio-system
  labels:
    istio/multiCluster: "true"
  annotations:
    networking.istio.io/cluster: cluster1
stringData:
  cluster1: |
    apiVersion: v1
    kind: Config
    clusters:
    - name: cluster1
      cluster:
        server: ${SERVER_CLUSTER1}
        certificate-authority-data: ${CA_B64_CLUSTER1}
    contexts:
    - name: cluster1
      context:
        cluster: cluster1
        user: cluster1
    current-context: cluster1
    users:
    - name: cluster1
      user:
        token: ${TOKEN_CLUSTER1}
EOF

# Create service account token for cluster2
kubectl apply --context kind-cluster2 -f - <<'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: istio-reader-service-account-istio-remote-secret-token
  namespace: istio-system
  annotations:
    kubernetes.io/service-account.name: istio-reader-service-account
type: kubernetes.io/service-account-token
EOF

# Wait for token generation
until kubectl get --context kind-cluster2 -n istio-system secret istio-reader-service-account-istio-remote-secret-token \
  -o jsonpath='{.data.token}' 2>/dev/null | grep -q .; do
  sleep 1
done

TOKEN_CLUSTER2="$(kubectl get --context kind-cluster2 -n istio-system secret istio-reader-service-account-istio-remote-secret-token \
  -o jsonpath='{.data.token}' | b64dec)"
CA_B64_CLUSTER2="$(kubectl get --context kind-cluster2 -n istio-system secret istio-reader-service-account-istio-remote-secret-token \
  -o jsonpath='{.data.ca\.crt}')"

# Create secret in cluster1 to access cluster2
kubectl apply --context kind-cluster1 --server-side -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: istio-remote-secret-cluster2
  namespace: istio-system
  labels:
    istio/multiCluster: "true"
  annotations:
    networking.istio.io/cluster: cluster2
stringData:
  cluster2: |
    apiVersion: v1
    kind: Config
    clusters:
    - name: cluster2
      cluster:
        server: ${SERVER_CLUSTER2}
        certificate-authority-data: ${CA_B64_CLUSTER2}
    contexts:
    - name: cluster2
      context:
        cluster: cluster2
        user: cluster2
    current-context: cluster2
    users:
    - name: cluster2
      user:
        token: ${TOKEN_CLUSTER2}
EOF

7. Create East-West Gateways

kubectl apply --context kind-cluster1 --server-side -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: istio-eastwestgateway
  namespace: istio-system
  labels:
    topology.istio.io/network: "network1"
spec:
  gatewayClassName: "istio-east-west"
  listeners:
    - name: mesh
      port: 15008
      protocol: HBONE
      tls:
        mode: Terminate
        options:
          gateway.istio.io/tls-terminate-mode: ISTIO_MUTUAL
EOF

kubectl apply --context kind-cluster2 --server-side -f - <<'EOF'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: istio-eastwestgateway
  namespace: istio-system
  labels:
    topology.istio.io/network: "network2"
spec:
  gatewayClassName: "istio-east-west"
  listeners:
    - name: mesh
      port: 15008
      protocol: HBONE
      tls:
        mode: Terminate
        options:
          gateway.istio.io/tls-terminate-mode: ISTIO_MUTUAL
EOF

Wait for EXTERNAL-IP assignment:

for c in kind-cluster1 kind-cluster2; do
  echo "=== waiting EXTERNAL-IP for $c"
  until kubectl --context "$c" -n istio-system get svc istio-eastwestgateway \
    -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null | grep -Eq '[0-9]'; do
    sleep 2
  done
  kubectl --context "$c" -n istio-system get svc istio-eastwestgateway
done

8. Verification

Create sample namespace and enable ambient mode:

kubectl create ns sample --context kind-cluster1
kubectl create ns sample --context kind-cluster2

kubectl label ns sample --context kind-cluster1 istio.io/dataplane-mode=ambient --overwrite
kubectl label ns sample --context kind-cluster2 istio.io/dataplane-mode=ambient --overwrite

Deploy helloworld (v1 to cluster1, v2 to cluster2):

# Create Service in both clusters
kubectl apply --context kind-cluster1 -n sample -f samples/helloworld/helloworld.yaml -l service=helloworld
kubectl apply --context kind-cluster2 -n sample -f samples/helloworld/helloworld.yaml -l service=helloworld

# Create Deployment with different versions in each cluster
kubectl apply --context kind-cluster1 -n sample -f samples/helloworld/helloworld.yaml -l version=v1
kubectl apply --context kind-cluster2 -n sample -f samples/helloworld/helloworld.yaml -l version=v2

# Mark as global service (enable cross-cluster communication)
kubectl label svc helloworld --context kind-cluster1 -n sample istio.io/global=true --overwrite
kubectl label svc helloworld --context kind-cluster2 -n sample istio.io/global=true --overwrite

Deploy curl:

kubectl apply --context kind-cluster1 -n sample -f samples/curl/curl.yaml
kubectl apply --context kind-cluster2 -n sample -f samples/curl/curl.yaml

Verify cross-cluster load balancing:

# Call helloworld from curl in cluster1
for i in {1..10}; do
  kubectl exec --context kind-cluster1 -n sample deploy/curl -c curl -- \
    curl -sS "helloworld.sample:5000/hello"
done

# Also verify from cluster2
for i in {1..10}; do
  kubectl exec --context kind-cluster2 -n sample deploy/curl -c curl -- \
    curl -sS "helloworld.sample:5000/hello"
done

Success if you see a mix of v1 and v2 responses.

Example:

$ for i in {1..10}; do
  kubectl exec --context kind-cluster1 -n sample deploy/curl -c curl -- \
    curl -sS "helloworld.sample:5000/hello"
done
Hello version: v1, instance: helloworld-v1-696f8879d6-w6g89
Hello version: v2, instance: helloworld-v2-59fc9f4558-g8r8b
Hello version: v1, instance: helloworld-v1-696f8879d6-w6g89
Hello version: v2, instance: helloworld-v2-59fc9f4558-g8r8b
Hello version: v1, instance: helloworld-v1-696f8879d6-w6g89
Hello version: v2, instance: helloworld-v2-59fc9f4558-g8r8b
Hello version: v1, instance: helloworld-v1-696f8879d6-w6g89
Hello version: v2, instance: helloworld-v2-59fc9f4558-g8r8b
Hello version: v2, instance: helloworld-v2-59fc9f4558-g8r8b
Hello version: v2, instance: helloworld-v2-59fc9f4558-g8r8b

Common Pitfalls

cloud-provider-kind not running: EXTERNAL-IP remains <pending> and never gets assigned
CA mismatch: Errors like curl: (56) Recv failure: Connection reset by peer occur
Missing istio.io/global=true label: Cross-cluster communication won't happen unless applied to Services in both clusters
Running outside the Istio repository: Errors occur when samples/ or manifests/ directories are not found

References

Extending Knative Service with Envoy Gateway Integration

Kahiro Okina — Tue, 23 Dec 2025 01:51:15 +0000

Introduction

This article demonstrates how to run Knative Serving in combination with Envoy Gateway and how to extend its functionality.

Prerequisites

The following tools must be installed:

kind
kubectl
helm
ko
Go

Installing cloud-provider-kind

cloud-provider-kind is a tool that enables LoadBalancer Services in kind clusters. Install it with the following command:

go install sigs.k8s.io/cloud-provider-kind@latest

Installing ko

ko is a tool for building and deploying applications written in Go as container images. Install it with the following command:

go install github.com/google/ko@latest

Setup Steps

1. Create a kind Cluster

First, create a local Kubernetes cluster with kind.

kind create cluster

2. Install Knative Serving

Install the Knative Serving CRDs and core components.

kubectl apply -f https://github.com/knative/serving/releases/latest/download/serving-crds.yaml
kubectl apply -f https://github.com/knative/serving/releases/latest/download/serving-core.yaml

3. Configure Knative Serving for Gateway API

Modify the Knative Serving network configuration to use Gateway API.

kubectl patch configmap/config-network \
  -n knative-serving \
  --type merge \
  -p '{"data":{"ingress.class":"gateway-api.ingress.networking.knative.dev"}}'

4. Create values File for Envoy Gateway

Create a configuration file for Envoy Gateway. This configuration deploys Envoy Proxy in the same namespace where the Gateway is deployed.

cat <<'YAML' > values-eg.yaml
config:
  envoyGateway:
    provider:
      type: Kubernetes
      kubernetes:
        deploy:
          type: GatewayNamespace
YAML

5. Install Envoy Gateway

Install Envoy Gateway using Helm.

export ENVOY_GATEWAY_VERSION=v1.6.1
helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version $ENVOY_GATEWAY_VERSION \
  -n envoy-gateway-system --create-namespace \
  -f ./values-eg.yaml

6. Start cloud-provider-kind

To enable LoadBalancer Services, open a separate terminal and start cloud-provider-kind. This process needs to remain running.

# Run in a separate terminal
sudo cloud-provider-kind

Leave this terminal running and return to the original terminal to continue with the following steps.

7. Clone the net-gateway-api Repository

Clone the repository containing resources for integrating Knative Serving with Gateway API.

git clone https://github.com/knative-extensions/net-gateway-api.git
cd net-gateway-api

8. Deploy Gateway API Integration Components

Use ko to deploy the components that integrate Knative Serving with Gateway API.

export KO_DOCKER_REPO=kind.local
ko apply -f config/

Note: If you created your KinD cluster with the --name option (e.g., kind create cluster --name my-cluster), you need to additionally set the KIND_CLUSTER_NAME environment variable as follows:

export KO_DOCKER_REPO=kind.local
export KIND_CLUSTER_NAME=my-cluster
ko apply -f config/

9. Deploy Gateway API Resources

Deploy the Gateway resources for Knative Serving.

kubectl apply -f ./third_party/envoy-gateway

10. Apply Gateway Configuration ConfigMap

Add configuration so that Knative Serving recognizes Envoy Gateway.

cat <<'YAML' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-gateway
  namespace: knative-serving
  labels:
    app.kubernetes.io/component: net-gateway-api
    app.kubernetes.io/name: knative-serving
    serving.knative.dev/release: devel
data:
  external-gateways: |
    - class: eg-external
      gateway: eg-external/eg-external
      service: eg-external/knative-external
      supported-features:
      - HTTPRouteRequestTimeout

  # local-gateways defines the Gateway to be used for cluster local traffic
  local-gateways: |
    - class: eg-internal
      gateway: eg-internal/eg-internal
      service: eg-internal/knative-internal
      supported-features:
      - HTTPRouteRequestTimeout
YAML

11. Configure Domain

Set up a test domain configuration.

kubectl patch configmap config-domain -n knative-serving --type merge -p '{"data":{"example.com":""}}'

12. Deploy Sample Application

Deploy a sample application for testing.

cat <<'YAML' | kubectl apply -f -
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-go
spec:
  template:
    spec:
      containers:
      - image: gcr.io/knative-samples/helloworld-go
        env:
        - name: TARGET
          value: Go Sample v1
YAML

Verification

1. Get LoadBalancer IP and Configure /etc/hosts

Get the LoadBalancer IP address.

export LB_IP=$(kubectl -n eg-external get svc knative-external -o jsonpath='{.status.loadBalancer.ingress[*].ip}{"\n"}')
echo $LB_IP

Add the obtained IP address to /etc/hosts.

echo "$LB_IP helloworld-go.default.example.com" | sudo tee -a /etc/hosts

2. Access Test

curl http://helloworld-go.default.example.com

If deployed successfully, you should receive a response like:

Hello Go Sample v1!

Advanced: Extending Knative Service with Envoy Gateway

One advantage of using Envoy Gateway is the ability to add functionality using resources like SecurityPolicy without modifying the Knative Service itself. Here's an example of adding Basic authentication.

Adding Basic Authentication

1. Create Authentication Credentials

First, create an authentication file with the htpasswd command (username: foo, password: bar).

htpasswd -cbs .htpasswd foo bar

2. Create Secret

kubectl -n default create secret generic basic-auth --from-file=.htpasswd

3. Apply SecurityPolicy

Use Envoy Gateway's SecurityPolicy resource to add Basic authentication to the HTTPRoute.

cat <<'YAML' | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: helloworld-go-basic-auth
  namespace: default
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: helloworld-go.default.example.com
  basicAuth:
    users:
      name: basic-auth
YAML

4. Verification

Accessing without authentication returns a 401 error.

curl http://helloworld-go.default.example.com
# 401 Unauthorized

Accessing with authentication credentials returns a successful response.

curl -u foo:bar http://helloworld-go.default.example.com
# Hello Go Sample v1!

This demonstrates how Basic authentication can be easily added using Envoy Gateway's SecurityPolicy without modifying the Knative Service or HTTPRoute manifests at all. Similarly, various features such as rate limiting, CORS configuration, JWT verification, etc., can be added at the Gateway layer, so please refer to Security | Envoy Gateway.

Summary

This article introduced how to run Knative Serving in combination with Envoy Gateway.

Envoy Gateway is a high-performance official Envoy gateway implementation, and its combination with Knative Serving enables flexible control and extension of serverless application routing.

Reference Links

Notes

HTTPRoute Output When Traffic Splitting is Enabled

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  annotations:
    networking.internal.knative.dev/rollout: '{"configurations":[{"configurationName":"helloworld-go","percent":50,"revisions":[{"revisionName":"helloworld-go-00005","percent":50}],"stepParams":{}},{"configurationName":"helloworld-go","tag":"qa","percent":100,"revisions":[{"revisionName":"helloworld-go-00005","percent":100}],"stepParams":{}}]}'
    networking.knative.dev/ingress.class: gateway-api.ingress.networking.knative.dev
    serving.knative.dev/creator: kubernetes-admin
    serving.knative.dev/lastModifier: kubernetes-admin
  creationTimestamp: "2025-11-09T05:41:51Z"
  generation: 33
  labels:
    networking.knative.dev/visibility: ""
    serving.knative.dev/route: helloworld-go
    serving.knative.dev/routeNamespace: default
    serving.knative.dev/service: helloworld-go
  name: helloworld-go.default.example.com
  namespace: default
  ownerReferences:
  - apiVersion: networking.internal.knative.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Ingress
    name: helloworld-go
    uid: f15718d7-91ba-482f-8322-27968a218697
  resourceVersion: "220984"
  uid: 07fca6bb-1e7a-4a53-8e09-107df56fcb25
spec:
  hostnames:
  - helloworld-go.default.example.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg-external
    namespace: eg-external
  rules:
  - backendRefs:
    - filters:
      - requestHeaderModifier:
          set:
          - name: Knative-Serving-Namespace
            value: default
          - name: Knative-Serving-Revision
            value: helloworld-go-00003
        type: RequestHeaderModifier
      group: ""
      kind: Service
      name: helloworld-go-00003
      port: 80
      weight: 50
    - filters:
      - requestHeaderModifier:
          set:
          - name: Knative-Serving-Namespace
            value: default
          - name: Knative-Serving-Revision
            value: helloworld-go-00005
        type: RequestHeaderModifier
      group: ""
      kind: Service
      name: helloworld-go-00005
      port: 80
      weight: 50
    filters:
    - requestHeaderModifier:
        set:
        - name: K-Network-Hash
          value: f3149388225bc6465417757efc25ea3b739741e2037cf7a5fc49772260fc882f
      type: RequestHeaderModifier
    matches:
    - headers:
      - name: K-Network-Hash
        type: Exact
        value: override
      path:
        type: PathPrefix
        value: /
    timeouts:
      request: 0s
  - backendRefs:
    - filters:
      - requestHeaderModifier:
          set:
          - name: Knative-Serving-Namespace
            value: default
          - name: Knative-Serving-Revision
            value: helloworld-go-00003
        type: RequestHeaderModifier
      group: ""
      kind: Service
      name: helloworld-go-00003
      port: 80
      weight: 50
    - filters:
      - requestHeaderModifier:
          set:
          - name: Knative-Serving-Namespace
            value: default
          - name: Knative-Serving-Revision
            value: helloworld-go-00005
        type: RequestHeaderModifier
      group: ""
      kind: Service
      name: helloworld-go-00005
      port: 80
      weight: 50
    matches:
    - path:
        type: PathPrefix
        value: /
    timeouts:
      request: 0s
status:
  parents:
  - conditions:
    - lastTransitionTime: "2025-11-10T04:00:42Z"
      message: Route is accepted
      observedGeneration: 33
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2025-11-10T04:00:42Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 33
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: eg-external
      namespace: eg-external

HTTPRoute Output When Using DomainMapping

Apply the following:

apiVersion: networking.internal.knative.dev/v1alpha1
kind: ClusterDomainClaim
metadata:
  name: test.com
spec:
  namespace: default
---
apiVersion: serving.knative.dev/v1beta1
kind: DomainMapping
metadata:
  name: test.com
  namespace: default
spec:
  ref:
    name: helloworld-go
    kind: Service
    apiVersion: serving.knative.dev/v1

The following is output:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  annotations:
    networking.knative.dev/ingress.class: gateway-api.ingress.networking.knative.dev
    serving.knative.dev/creator: kubernetes-admin
    serving.knative.dev/lastModifier: kubernetes-admin
  creationTimestamp: "2025-11-10T04:48:55Z"
  generation: 1
  labels:
    networking.knative.dev/visibility: ""
    serving.knative.dev/domainMappingNamespace: default
    serving.knative.dev/domainMappingUID: 88954e71-03c4-4004-81e1-2916c6ba976c
  name: test.com
  namespace: default
  ownerReferences:
  - apiVersion: networking.internal.knative.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Ingress
    name: test.com
    uid: d7351ea1-9e67-4ebc-80cb-4227195362c1
  resourceVersion: "231722"
  uid: 45f70e96-0f30-4cda-b1a2-f6f1a25dad0b
spec:
  hostnames:
  - test.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg-external
    namespace: eg-external
  rules:
  - backendRefs:
    - filters:
      - requestHeaderModifier:
          set:
          - name: K-Original-Host
            value: test.com
        type: RequestHeaderModifier
      group: ""
      kind: Service
      name: helloworld-go
      port: 80
      weight: 100
    filters:
    - requestHeaderModifier:
        set:
        - name: K-Network-Hash
          value: 244b84148c16d15938fd7f3ab2d377b0b40f6418229edee9af3fe0d961fd152a
      type: RequestHeaderModifier
    - type: URLRewrite
      urlRewrite:
        hostname: helloworld-go.default.svc.cluster.local
    matches:
    - headers:
      - name: K-Network-Hash
        type: Exact
        value: override
      path:
        type: PathPrefix
        value: /
    timeouts:
      request: 0s
  - backendRefs:
    - filters:
      - requestHeaderModifier:
          set:
          - name: K-Original-Host
            value: test.com
        type: RequestHeaderModifier
      group: ""
      kind: Service
      name: helloworld-go
      port: 80
      weight: 100
    filters:
    - type: URLRewrite
      urlRewrite:
        hostname: helloworld-go.default.svc.cluster.local
    matches:
    - path:
        type: PathPrefix
        value: /
    timeouts:
      request: 0s
status:
  parents:
  - conditions:
    - lastTransitionTime: "2025-11-10T04:48:55Z"
      message: Route is accepted
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2025-11-10T04:48:55Z"
      message: Resolved all the Object references for the Route
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: eg-external
      namespace: eg-external

This helloworld-go.default.svc.cluster.local points to the Envoy Proxy of the internal Envoy Gateway.

apiVersion: v1
kind: Service
metadata:
  annotations:
    serving.knative.dev/creator: kubernetes-admin
    serving.knative.dev/lastModifier: kubernetes-admin
  creationTimestamp: "2025-11-09T05:41:51Z"
  labels:
    serving.knative.dev/route: helloworld-go
    serving.knative.dev/service: helloworld-go
  name: helloworld-go
  namespace: default
  ownerReferences:
  - apiVersion: serving.knative.dev/v1
    blockOwnerDeletion: true
    controller: true
    kind: Route
    name: helloworld-go
    uid: 24cc3f82-f7b3-40d9-89ce-ea516e46382f
  resourceVersion: "56240"
  uid: c6ea6306-abf9-49c0-8a28-9a71c9a3a99a
spec:
  externalName: knative-internal.eg-internal.svc.cluster.local
  ports:
  - appProtocol: kubernetes.io/h2c
    name: http2
    port: 80
    protocol: TCP
    targetPort: 80
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}

Therefore, for example, when exposed via ALB, the path from external traffic would be as follows, which includes one additional hop through the Internal Envoy Proxy compared to the typical path exposed through External Envoy:

flowchart LR
  A["ALB"]

  subgraph K8S["Kubernetes"]
    direction LR

    subgraph EXT["Namespace: eg-external"]
      B["External Envoy Proxy"]
    end

    subgraph INT["Namespace: eg-internal"]
      C["Internal Envoy Proxy"]
    end

    subgraph KS["Namespace: knative-serving"]
      D["Knative Activator"]
    end

    subgraph P["Knative Revision Pod"]
      direction LR
      E["Queue Proxy (sidecar)"]
      F["User App Container"]
    end
  end

  A e1@==> B
  B e2@==> C
  C e3@==> D
  D e4@==> E
  E e5@==> F

  e1@{ animate: true }
  e2@{ animate: true }
  e3@{ animate: true }
  e4@{ animate: true }
  e5@{ animate: true }

[Gateway API] How to Extend Existing HTTPRoute Without Modification Using GEP-1294

Kahiro Okina — Thu, 23 Oct 2025 01:14:03 +0000

Prerequisites

Please use a Service Mesh such as Istio that implements GEP-1294: xRoutes Mesh Binding.

Introduction

Without using Shadow Service, you can extend existing HTTPRoute without touching a single line of YAML using only Gateway API standard features. The key point is the specification that allows you to specify a Kubernetes Service in the parentRefs of HTTPRoute (GEP-1294).

According to GEP-1294's Route presence, when you create an xRoute with a Service as the parentRef, the implicit delivery (normal Service→Endpoints routing) for that Service is not "added" but can be replaced by the Route. Furthermore, Namespace boundaries states that a producer route with a Service in the same Namespace as parentRef SHOULD be applied to all inbound requests to that Service.

Benefits

You can customize even when existing tools don't provide fine-grained HTTPRoute customization
Can be achieved using only Gateway API specifications

Illustrated Diagram

Show Mermaid

flowchart LR
  classDef parent stroke-dasharray: 4 2;
  classDef note fill:#FFF9C4,stroke:#FBC02D,color:#5D4037,stroke-width:1px;

  subgraph Ingress
    GW[Gateway]
  end

  subgraph app-ns
    HR_a["HTTPRoute a (existing)"]
    SVC_a[(Service a)]
    DEP_a[Deployment a]
  end

  subgraph ext-ns
    HR_b["HTTPRoute b (new)"]
    SVC_b[(Service b)]
    DEP_b[Deployment b]
  end

  GW e1@--&gt; HR_a
  HR_a -.parent ref: registration.-&gt; GW
  HR_a e2@--&gt; SVC_a
  SVC_a -."selector: Note 1".-x DEP_a
  SVC_a e4@--&gt; HR_b
  HR_b -."parentRef: override".-&gt; SVC_a
  HR_b e5@--&gt; SVC_b
  SVC_b e6@--&gt; DEP_b

  subgraph Notes
    direction TB
    N1["Note 1: A Route that uses a Service as parentRef **replaces** the implicit routing of that Service (GEP-1294 'Route presence'). Therefore, the selector on the Service side does not affect the final destination determination."]:::note
  end

  e1@{ animate: true }
  e2@{ animate: true }
  e4@{ animate: true }
  e5@{ animate: true }
  e6@{ animate: true }

Use Cases

Without modifying HTTPRoute managed by a Controller, you can switch the destination Service or add necessary L7 controls (match/header manipulation/retry/split, etc.) using another HTTPRoute with Service as parent
"I want to control all inbound traffic to that Service" → producer route
"I want to redirect only outbound traffic from my Namespace" → consumer route

Additional Explanation

In GEP-1294, the meaning changes based on the relationship between the Namespace of the Service pointed to by parentRef and the Namespace of the Route. There is no explicit field in the API; this is purely a distinction in behavior.

This enables the following scenarios:

When you want to control the entire ingress point, define a producer route on the target Service side
When you want to control only outbound traffic from your Namespace, create a consumer route in a different Namespace

In both cases, this can be achieved by adding routes without touching existing HTTPRoute.

producer route (control ingress side)

Application conditions:
When parentRef points to a Service in the same Namespace

Behavior:

SHOULD be applied to all inbound requests to that Service (including calls from other Namespaces)
Route rules take precedence over the Service's default routing
Note: Some implementations apply this at the client-side proxy, so it may not affect clients outside the mesh

The diagram shown earlier illustrates this producer route.

consumer route (control egress side)

Application conditions:
When parentRef points to a Service in a different Namespace

Behavior:

Applied only to egress traffic from within the Namespace where the Route exists (limited scope)
Provides boundaries to prevent unrelated third-party Namespaces from arbitrarily changing traffic between different Namespaces

① A certain namespace has HTTPRoute a applied (consumer route / normal path)

flowchart LR
  classDef note fill:#FFF9C4,stroke:#FBC02D,color:#5D4037,stroke-width:1px;

  subgraph app-ns
    SVC_a[(Service a)]
    DEP_a[Deployment a]
  end

  subgraph other-ns["other-ns (e.g., some ns)"]
    CALL_other["Pod/Sidecar (caller)"]
    HR_a["HTTPRoute a (consumer)"]
  end

  CALL_other e1@--&gt; HR_a
  HR_a -."parentRef: app-ns / Service a (different NS ↔ consumer)".-&gt; SVC_a
  HR_a e2@--&gt; SVC_a
  SVC_a e3@--&gt; DEP_a

  e1@{ animate: true }
  e2@{ animate: true }
  e3@{ animate: true }

② A specific namespace has HTTPRoute b applied (consumer route / destination redirect)

flowchart LR
  classDef note fill:#FFF9C4,stroke:#FBC02D,color:#5D4037,stroke-width:1px;

  subgraph app-ns
    SVC_a[(Service a)]
  end

  subgraph ext-ns
    SVC_b[(Service b)]
    DEP_b[Deployment b]
  end

  subgraph dev-ns["dev-ns (specific ns)"]
    CALL_dev["Pod/Sidecar (caller)"]
    HR_b["HTTPRoute b (consumer)"]
  end

  CALL_dev e1@--&gt; HR_b
  HR_b -."parentRef: app-ns / Service a (different NS ↔ consumer)".-&gt; SVC_a
  HR_b e2@--&gt; SVC_b
  SVC_b e3@--&gt; DEP_b

  e1@{ animate: true }
  e2@{ animate: true }
  e3@{ animate: true }

Example

Set up HTTPRoute normally with Gateway API

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gw
  namespace: app-ns
spec:
  gatewayClassName: istio
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      hostname: example.local
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: route-a
  namespace: app-ns
spec:
  hostnames:
    - example.local
  parentRefs:
    - name: gw
      namespace: app-ns
  rules:
    - backendRefs:
        - kind: Service
          name: svc-a
          port: 80
          weight: 100
---
apiVersion: v1
kind: Service
metadata:
  name: svc-a
  namespace: app-ns
spec:
  selector:
    app: a
  ports:
    - name: http
      port: 80
      targetPort: 5678

Without modifying the above HTTPRoute or Service, override the entire ingress point with a producer route (place route-b in app-ns)

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: route-b
  namespace: app-ns  # Same NS, so this is a producer route
spec:
  parentRefs:
    - group: ""
      kind: Service
      name: svc-a
      namespace: app-ns
      port: 80
  rules:
    - backendRefs:
        - kind: Service
          name: svc-b
          namespace: ext-ns  # Switch to destination in different NS
          port: 80
          weight: 100
---
apiVersion: v1
kind: Service
metadata:
  name: svc-b
  namespace: ext-ns
spec:
  selector:
    app: b
  ports:
    - name: http
      port: 80
      targetPort: 5678

Since the above is a producer route (parent reference in the same NS), the rules of route-b are applied to all inbound requests to svc-a, and the destination is redirected to svc-b (depending on the mesh implementation, the application point may be on the client side).

Note on ReferenceGrant

GEP-1294 explicitly states that in the Mesh (East/West) context, Routes with Service as parentRef can reference backendRefs in different Namespaces and are allowed without ReferenceGrant:

https://github.com/kubernetes-sigs/gateway-api/blob/main/geps/gep-1294/index.md#route-presence:~:text=Routes%20of%20either,a%20broader%20scope

Routes of either type can send traffic to backendRefs in any namespace. Unlike Gateway bound routes, this is allowed without a ReferenceGrant. In Gateway-bound routes (North-South), routes are opt-in; by default, no Services are exposed (often to the public internet), and a service producer must explicitly opt-in by creating a route themselves, or allowing another namespace to via ReferenceGrant. For mesh, routes augment existing Services, rather than exposing them to a broader scope.

Additionally, it notes that access control should be handled by mechanisms such as NetworkPolicy:

https://github.com/kubernetes-sigs/gateway-api/blob/main/geps/gep-1294/index.md#route-presence:~:text=As%20a%20result%2C%20a%20ReferenceGrant%20is%20not%20required%20in%20most%20mesh%20implementations.%20Access%20control%2C%20if%20desired%2C%20is%20handled%20by%20other%20mechanism%20such%20as%20NetworkPolicy

As a result, a ReferenceGrant is not required in most mesh implementations. Access control, if desired, is handled by other mechanism such as NetworkPolicy

Therefore, the configuration in this article where a producer route in app-ns routes to svc-b in ext-ns does not require ReferenceGrant.

Summary

I introduced a method to change the reference destination of HTTPRoute generated by other controllers using GEP-1294, achieving the same result as if you had written additional configuration in the HTTPRoute, when you cannot customize that HTTPRoute directly.

I hope this serves as a reference for someone.

Reference Links

Route presence: https://gateway-api.sigs.k8s.io/geps/gep-1294/#route-presence
Namespace boundaries (producer / consumer boundaries): https://gateway-api.sigs.k8s.io/geps/gep-1294/#namespace-boundaries

Shadow Service Pattern

Kahiro Okina — Thu, 23 Oct 2025 01:13:47 +0000

Shadow Service Pattern

The Shadow Service pattern is a mechanism for communication using a Service on Kubernetes that has "no actual destination." This guide explains the specific method of creating a Service with an assigned ClusterIP (Shadow Service) and the EndpointSlice that the Service references. The key feature is that it can be completed using only standard Kubernetes features, without needing to change kube-proxy settings.

What is a Shadow Service?

A Service that is linked to an arbitrary IP rather than referencing Pods running on Nodes
Utilizes the Kubernetes Service object mechanism as-is, and ClusterIP can be automatically assigned
Can be implemented without modifying default Kubernetes components like kube-proxy

By creating what is essentially a "dummy" Service and defining the endpoint information (such as IP and port) that becomes the actual backing for that Service yourself using EndpointSlice, you can make "external resources" accessible via ClusterIP within Kubernetes.

Implementation Flow

Create the Shadow Service
Create the EndpointSlice
Associate the Service name with the EndpointSlice label
Access external resources via the ClusterIP

1. Creating the Shadow Service

Since the Shadow Service doesn't have Pods as a backend, you don't specify or disable the selector and type (NodePort, LoadBalancer, etc.) in a typical Service definition. In the following example, the name is set to "dummy" and HTTPS (443) port is defined.

apiVersion: v1
kind: Service
metadata:
  name: dummy
  namespace: default
spec:
  ports:
    - port: 443
      protocol: TCP
      targetPort: 443
      name: https

At this stage, when you execute kubectl apply -f shadow-service.yaml, a ClusterIP for the Service will be automatically assigned.

2. Creating the EndpointSlice

Next, define an EndpointSlice as the endpoint linked to the Shadow Service.

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: dummy
  namespace: default
  labels:
    kubernetes.io/service-name: dummy
addressType: IPv4
ports:
- name: https
  protocol: TCP
  port: 443
endpoints:
- addresses:
  - "93.184.215.14" # Example: IP of example.com, specify external endpoint
  conditions:
    ready: true
  zone: zone-a
  nodeName: node-a

In the above, "93.184.215.14" is linked to the endpoint as an external resource. By specifying dummy in the kubernetes.io/service-name label, it is associated with the Service dummy created earlier.

3. Association

If the Shadow Service's metadata.name and the EndpointSlice's labels.kubernetes.io/service-name are the same, they are automatically associated. This connects the Service entry point (ClusterIP and port 443) with the actual external endpoint ("93.184.215.14").

4. Access Overview

Once complete, requests to the ClusterIP (e.g., 10.96.0.X) will be forwarded to the external IP specified in the EndpointSlice. From the application or user's perspective, it appears as if they are accessing a Service called "dummy," but the actual backing is an external address.

Testing

You can test access using curl as follows:

curl https://dummy --insecure -H "Host: www.example.com"

Benefits and Considerations

Benefits

Completed within the scope of existing Kubernetes components
Can obtain a ClusterIP even without having Pods

Considerations

Manage the availability and validity of the IP addresses specified in the EndpointSlice
Verify whether other integrated Ingress Controllers or Gateway Controllers support such Shadow Services

References

Services without selectors | Kubernetes

No More Self-Building Required! CoreDNS v1.12.2 Now Includes Standard Multicluster Support

Kahiro Okina — Mon, 08 Sep 2025 05:27:53 +0000

TL;DR

Multicluster support has been integrated into CoreDNS v1.12.2!
- https://github.com/coredns/coredns/pull/7266
You can now resolve clusterset.local by just configuring the Corefile and RBAC

Background

Previously, it was necessary to self-build CoreDNS by adding the coredns/multicluster plugin, but starting with CoreDNS v1.12.2, it has been integrated into the Kubernetes plugin, allowing you to handle clusterset.local with just the official image.

Examples of Corefile configuration can be found in the official README.

Setup Method

Image: Use registry.k8s.io/coredns/coredns:v1.12.2 or later versions
Edit the Corefile

Corefile Configuration

Add clusterset.local to the kubernetes plugin and enable multicluster.

kubernetes cluster.local clusterset.local {
    multicluster clusterset.local
}

Command to directly patch the existing kube-system/coredns ConfigMap:

kubectl --kubeconfig "${KUBECONFIG}" get configmap -n kube-system coredns -o yaml | \
  sed -E 's/^([[:space:]]*)kubernetes cluster\.local (.*)$/\1kubernetes cluster.local clusterset.local \2\n\1   multicluster clusterset.local/' | \
  kubectl --kubeconfig "${KUBECONFIG}" replace -f-

Verification after application is recommended

Granting Permissions

Extend the ClusterRole so that CoreDNS can list/watch ServiceImport.

rules:
- apiGroups: ["multicluster.x-k8s.io"]
  resources: ["serviceimports"]
  verbs: ["list", "watch"]

Command to add permissions:

kubectl patch clusterrole system:coredns --type=json --patch '[
  {
    "op": "add",
    "path": "/rules/-",
    "value": {
      "apiGroups": ["multicluster.x-k8s.io"],
      "resources": ["serviceimports"],
      "verbs": ["list","watch"]
    }
  }
]'

Rollout

Update the CoreDNS image to v1.12.2.

kubectl -n kube-system set image deploy/coredns coredns=registry.k8s.io/coredns/coredns:v1.12.2
kubectl -n kube-system rollout status deploy/coredns

Verification Tips

Check if ServiceImport is visible

kubectl get serviceimports.multicluster.x-k8s.io -A

Check if clusterset.local can be resolved

From a debug pod or similar:

kubectl exec -it -n default deploy/your-app -- sh -c 'dig +short my-svc.my-namespace.svc.clusterset.local'

References & Extras

CoreDNS Kubernetes Plugin Examples: https://github.com/coredns/coredns/blob/master/plugin/kubernetes/README.md#examples
Demo: Clone https://github.com/kubernetes-sigs/mcs-api/pull/126 and run make demo

Introduction to KEP-5339: Plugin for Credentials in ClusterProfile

Kahiro Okina — Tue, 02 Sep 2025 00:02:28 +0000

Introduction
This article summarizes the key points of KEP-5339 "Plugin for Credentials in ClusterProfile" (SIG-Multicluster) and presents a simple implementation example using EKS.
KEP: https://github.com/kubernetes/enhancements/tree/master/keps/sig-multicluster/5339-clusterprofile-plugin-credentials

Project Introduction: ClusterProfile Credentials Plugins

To make ClusterProfile Credentials Plugins easily testable, I've released a repository containing ready-to-use plugins.

Repository: labthrust/clusterprofile-credentials-plugins
Included plugins (all implemented as single binaries in Go):
- eks-aws-auth-plugin: Resolves clusters from EKS endpoints and CA, returns ExecCredential
- secretreader-plugin: Reads data.token from Kubernetes Secrets, returns ExecCredential

This repository aims to become an Awesome Plugins style catalog. I plan to continue adding and introducing useful plugins, with SPIRE plugin being one consideration for the future.

The reason I started this project is that while Credentials Plugin is an excellent specification, there's currently a significant hurdle of "having to create your own plugin first." Additionally, information about which plugins are available, how to use them, and where to find the binaries is not well organized.
Therefore, I've released this as a "collection of plugins you can immediately deploy and try" to lower the barrier for taking the first steps with ClusterProfile.

Overview

ClusterProfile's status.credentialProviders[].cluster holds server / CA.
The controller registers exec plugins in cp-creds.json and obtains tokens using KUBERNETES_EXEC_INFO passed at runtime as input.
Using .spec.cluster.server as the starting point, it identifies EKS clusters and calls aws eks get-token.
Input and output are in ExecCredential format.

Overall Flow (Mermaid)

sequenceDiagram
  autonumber
  participant Ctrl as Controller
  participant Prov as Credential Provider (cp-creds.json)
  participant Plugin as exec plugin<br/>(./eks-aws-auth-plugin.sh)
  participant AWS as AWS EKS API

  Ctrl->>Prov: Load providers
  Ctrl->>Plugin: exec providers[].execConfig.command<br/>env: KUBERNETES_EXEC_INFO
  Note right of Plugin: Extract region from server<br/>List EKS → resolve clusterName by exact endpoint match

  Plugin->>AWS: eks list-clusters (region)
  Plugin->>AWS: eks describe-cluster (each cluster's endpoint)
  AWS-->>Plugin: endpoint list
  Plugin->>AWS: eks get-token (clusterName)
  AWS-->>Plugin: ExecCredential(JSON)<br/>status.token, expirationTimestamp

  Plugin-->>Ctrl: Return ExecCredential(JSON) via stdout

Hands-On Example

Complete sample: https://github.com/kahirokunn/cluster-inventory-api/tree/eks-example

1) Start Controller

go build  controller_example.go
./controller_example -clusterprofile-provider-file ./cp-creds.json

2) Contents of cp-creds.json

This is the contents of the JSON file passed with the -clusterprofile-provider-file flag:

{
  "providers": [
    {
      "name": "eks",
      "execConfig": {
          "apiVersion": "client.authentication.k8s.io/v1beta1",
          "args": null,
          "command": "./eks-aws-auth-plugin.sh",
          "env": null,
          "provideClusterInfo": true
      }
    }
  ]
}

3) ClusterProfile YAML State

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ClusterProfile
metadata:
  name: my-cluster-1
spec:
  displayName: my-cluster-1
  clusterManager:
    name: EKS-Fleet
status:
  credentialProviders:
  - name: eks
    cluster:
      server: https://xxx.gr7.ap-northeast-1.eks.amazonaws.com
      certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1J...

ExecCredential and exec plugin Flow

ExecCredential
Kubernetes exec authentication plugin protocol. At runtime, input is provided via the KUBERNETES_EXEC_INFO environment variable, and output returns ExecCredential to stdout.
Specification: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#input-and-output-formats
Usage with kubectl / client-go
When you configure a command in kubeconfig's users[].exec or in credentials provider's providers[].execConfig as in this article, the command is launched during authentication. KUBERNETES_EXEC_INFO is passed to the plugin at that time, and kubectl/client-go consumes the ExecCredential from the plugin's stdout.
Current Configuration
./eks-aws-auth-plugin.sh is specified in providers[].execConfig.command. The plugin uses .spec.cluster.server and CA from KUBERNETES_EXEC_INFO as clues to identify the cluster and ultimately outputs ExecCredential.

Background: "Compatibility" of `aws eks update-kubeconfig` / `aws eks get-token`

aws eks update-kubeconfig
- Function: Command that generates/updates local kubeconfig.
- Input: Requires explicit cluster name specification via --name, etc. No mechanism to receive KUBERNETES_EXEC_INFO (ExecCredential's spec.cluster.server / CA).
- Output: Does not return ExecCredential JSON to stdout. Only modifies kubeconfig files. → Conclusion: Not compatible with exec plugin input/output requirements of "input = KUBERNETES_EXEC_INFO, output = ExecCredential(JSON)/stdout".
aws eks get-token
- Output: Returns ExecCredential-compatible JSON to stdout.
- Input: Requires explicit specification of --cluster-name. Cannot directly receive KUBERNETES_EXEC_INFO as input (lacks functionality to resolve cluster names from just server/CA). → Conclusion: Output is compatible, but input is not. A separate resolution layer from KUBERNETES_EXEC_INFO (server/CA) → cluster-name is needed.

EKS exec plugin Implementation

Processing flow:

Get .spec.cluster.server from KUBERNETES_EXEC_INFO
Extract region from hostname
List EKS in that region and identify cluster name by exact endpoint match
Execute aws eks get-token

#!/usr/bin/env bash
set -euo pipefail

# -------- utils --------
err() { printf "[eks-exec-credential] %s\n" "$*" >&2; }
need() { command -v "$1" >/dev/null 2>&1 || { err "missing dependency: $1"; exit 1; }; }
normalize_host() { sed -E 's#^https?://##; s#/$##; s#:443$##'; }

need jq
need aws

# --- read ExecCredential ---
if [[ -z "${KUBERNETES_EXEC_INFO:-}" ]]; then
  err "KUBERNETES_EXEC_INFO is empty. set provideClusterInfo: true"
  exit 1
fi

REQ_API_VERSION="$(jq -r '.apiVersion // empty' <<<"$KUBERNETES_EXEC_INFO")"
SERVER="$(jq -r '.spec.cluster.server // empty' <<<"$KUBERNETES_EXEC_INFO")"
if [[ -z "$SERVER" || "$SERVER" == "null" ]]; then
  err "spec.cluster.server is missing in KUBERNETES_EXEC_INFO"
  exit 1
fi

NORM_SERVER="$(printf "%s" "$SERVER" | normalize_host)"

# --- region: infer from server hostname ---
HOST="${NORM_SERVER%%/*}"
REGION="$(printf "%s\n" "$HOST" \
  | sed -nE 's#.*\.([a-z0-9-]+)\.eks(-fips)?\.amazonaws\.com(\.cn)?$#\1#p')"
if [[ -z "$REGION" ]]; then
  err "failed to parse region from server hostname: ${SERVER}"
  err "expected something like ...<random>.<suffix>.<region>.eks.amazonaws.com"
  exit 1
fi

# --- tiny cache: endpoint -> cluster name ---
CACHE_DIR="${XDG_CACHE_HOME:-$HOME/.cache}/eks-exec-credential"
mkdir -p "$CACHE_DIR"
MAP_CACHE="$CACHE_DIR/endpoint-map-${REGION}.json"
if [[ ! -s "$MAP_CACHE" ]] || ! jq -e . >/dev/null 2>&1 <"$MAP_CACHE"; then
  echo '{}' >"$MAP_CACHE"
fi

lookup_cache() { jq -r --arg k "$NORM_SERVER" '.[$k] // empty' <"$MAP_CACHE"; }
update_cache() {
  local tmp; tmp="$(mktemp)"
  jq --arg k "$NORM_SERVER" --arg v "$1" '.[$k]=$v' "$MAP_CACHE" >"$tmp" && mv "$tmp" "$MAP_CACHE"
}

match_endpoint() {
  local name="$1"
  local ep norm_ep
  ep="$(aws eks describe-cluster --region "$REGION" --name "$name" \
        --query 'cluster.endpoint' --output text 2>/dev/null || true)"
  [[ -z "$ep" || "$ep" == "None" ]] && return 1
  norm_ep="$(printf "%s" "$ep" | normalize_host)"
  [[ "$norm_ep" == "$NORM_SERVER" ]]
}

CLUSTER_NAME=""
# 1) cache hit?
CACHED="$(lookup_cache || true)"
if [[ -n "$CACHED" ]] && match_endpoint "$CACHED"; then
  CLUSTER_NAME="$CACHED"
fi

# 2) enumerate if needed
if [[ -z "$CLUSTER_NAME" ]]; then
  err "resolving cluster in ${REGION} for ${NORM_SERVER}"
  found=""
  while IFS= read -r name; do
    [[ -z "$name" ]] && continue
    if match_endpoint "$name"; then
      found="$name"
      break
    fi
  done < <(aws eks list-clusters --region "$REGION" --output json | jq -r '.clusters[]?')

  if [[ -z "$found" ]]; then
    err "no matching EKS cluster for endpoint: ${SERVER} (region=${REGION})"
    exit 1
  fi
  CLUSTER_NAME="$found"
  update_cache "$CLUSTER_NAME" || true
fi

# --- fetch ExecCredential via aws CLI ---
TOKEN_JSON="$(aws eks get-token --region "$REGION" --cluster-name "$CLUSTER_NAME" --output json)"

printf "%s\n" "$TOKEN_JSON"

Controller Usage Example

Sample code that generates rest.Config and creates a Clientset.

package main

import (
    "context"
    "encoding/base64"
    "flag"
    "log"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    clientcmdv1 "k8s.io/client-go/tools/clientcmd/api/v1"
    "sigs.k8s.io/cluster-inventory-api/apis/v1alpha1"
    "sigs.k8s.io/cluster-inventory-api/pkg/credentials"
)

func main() {
    credentialsProviders := credentials.SetupProviderFileFlag()
    flag.Parse()

    cpCreds, err := credentials.NewFromFile(*credentialsProviders)
    if err != nil {
        log.Fatalf("Got error reading credentials providers: %v", err)
    }

    caPEMBase64 := `LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1J...`
    caPEM, err := base64.StdEncoding.DecodeString(caPEMBase64)
    if err != nil {
        log.Fatalf("CA PEM base64 decode failed: %v", err)
    }

    // normally we would get this clusterprofile from the local cluster (maybe a watch?)
    // and we would maintain the restconfigs for clusters we're interested in.
    exampleClusterProfile := v1alpha1.ClusterProfile{
        Spec: v1alpha1.ClusterProfileSpec{
            DisplayName: "My Cluster",
        },
        Status: v1alpha1.ClusterProfileStatus{
            CredentialProviders: []v1alpha1.CredentialProvider{
                {
                    Name: "eks",
                    Cluster: clientcmdv1.Cluster{
                        Server: "https://xxx.gr7.ap-northeast-1.eks.amazonaws.com",
                        CertificateAuthorityData: caPEM,
                    },
                },
            },
        },
    }

    restConfigForMyCluster, err := cpCreds.BuildConfigFromCP(&exampleClusterProfile)
    if err != nil {
        log.Fatalf("Got error generating restConfig: %v", err)
    }
    log.Printf("Got credentials: %v", restConfigForMyCluster)
    // I can then use this rest.Config to build a k8s client.

    // Build a client and list Pods in the default namespace
    clientset, err := kubernetes.NewForConfig(restConfigForMyCluster)
    if err != nil {
        log.Fatalf("failed to create clientset: %v", err)
    }
    ctx := context.Background()
    pods, err := clientset.CoreV1().Pods("default").List(ctx, metav1.ListOptions{})
    if err != nil {
        log.Fatalf("failed to list pods: %v", err)
    }
    log.Printf("default namespace has %d pods", len(pods.Items))
    for i, p := range pods.Items {
        if i >= 10 {
            log.Printf("... (truncated)")
            break
        }
        log.Printf("pod: %s", p.Name)
    }
}

Troubleshooting

no matching EKS cluster → Check results, permissions, and profiles for aws eks list-clusters --region <r>
x509: certificate signed by unknown authority → Verify certificate-authority-data (base64)

Additional Notes

ExecCredential allows passing additional values using extensions.

KEP-5339: ClusterProfile's Credentials Plugin does not define specifications for using ExecCredential.extensions.
Related code: https://github.com/kubernetes-sigs/cluster-inventory-api/blob/main/pkg/credentials/config.go#L133

References

KEP-5339 (SIG-Multicluster) https://github.com/kubernetes/enhancements/tree/master/keps/sig-multicluster/5339-clusterprofile-plugin-credentials
ExecCredential Input and Output Formats https://kubernetes.io/docs/reference/access-authn-authz/authentication/#input-and-output-formats

🚀 Kubernetes v1.33's Image Volume β is on Fire! "Code-Only" Images are Changing the Game!

Kahiro Okina — Wed, 11 Jun 2025 02:32:20 +0000

Kubernetes v1.33 has finally brought Image Volume to β status! (※disabled by default)
This feature allows you to mount OCI images directly as read-only volumes - isn't this revolutionary?

🧠 What Makes Image Volume So Amazing?

Say goodbye to the old way of "cramming everything into one container", and hello to:

Application code
Runtime
Libraries

...all split into separate images and mounted independently!

This is especially powerful for scripting language users (Node.js/Python/etc.)!
Example: Mount a "code-only" image containing your app, and run it with node:22.15.1.

🧱 Building an Ultra-Lightweight "Code-Only" Image

The beauty of Image Volume is creating ultra-lightweight OCI images containing only code and config files.
Let's build a simple example with just Node.js application code.

📁 `index.js` – The App

const http = require('http');
http.createServer((_, res) => res.end('Hello from OCI volume!\n'))
    .listen(process.env.PORT || 80);

A simple HTTP server in Node.js that returns "Hello from OCI volume!"

🐳 `Dockerfile` – The Shocking "FROM scratch"

FROM scratch
COPY app /

Starting FROM scratch (= nothing) and just COPYing the code.
That's right - not even Node.js is included.

What does this mean? The runtime (Node.js) is provided by a separate container,
and this image is used purely as a "code repository."

📏 The Size? A Mere 134 Bytes!

This OCI image, composed of just these 3 files, weighs in at only 134 bytes!

docker images | grep hello-app
localhost:5001/hello-app                                                799466d0                 c4bd064f6fff   About a minute ago   134B

Builds in an instant ✅
Pushes to registry almost instantly ✅
Minimal CI/CD load ✅

As long as the code doesn't change, this image can be reused, improving cache hit rates.

This approach of "packaging only application code as an image" brings a completely different worldview
from traditional "all-in-one containers."

With Image Volume, you can completely separate the application code lifecycle from the runtime lifecycle.

📦 Instant Base Image Patches! See the Diff

With ImageVolume, following security patches for base images becomes surprisingly easy.

✅ Pattern 1: Fixed Tags with Manual Updates

For example, when a vulnerability patch is released for the node:22.15.0 base image:

-    image: mirror.gcr.io/node:22.15.0
+    image: mirror.gcr.io/node:22.15.1

Just update the Pod's image and redeploy - that's it!
No changes needed to the "app code" image!
No CI rebuilds needed. Just change one line of YAML, and security patches are applied instantly.

🔄 Pattern 2: Floating Tags (`node:22`) for Automatic Updates

For a more "fully automated" approach, use major version tags (node:22) instead of minor versions,
and set imagePullPolicy: Always. Patches are applied with just a Pod restart.

image: node:22
imagePullPolicy: Always

This way, updates from 22.15.0 → 22.15.1 → 22.16.0 are automatically applied with just Pod restarts.

:::caution ⚠️ Caution
While this method always incorporates the latest patches,
there's also the risk of immediately hitting regressions (bugs in new versions).
Careful consideration is needed for production environments where stability is paramount.

Explicit tags are safer for production.
:::

💡 Summary

Strategy	Benefits	Considerations
Fixed tags (`node:22.15.1`)	Stable, predictable behavior	Manual updates required
Floating tags + Pull Always (`node:22`)	Latest patches auto-applied	May upgrade to unintended versions

Choose a tag strategy that balances risk and automation based on your situation and environment.

🧪 Try It Now! Complete Automated Script Included

For those eager to experience the Image Volume world, here's a one-shot verification script:

✅ Full Verification Script:

#!/usr/bin/env bash
set -euox pipefail

# Create a temporary file for the upstream script
TMP_SCRIPT=$(mktemp)
echo "Downloading upstream kind-with-registry.sh to $TMP_SCRIPT"

# Download the upstream script
curl -fsSL https://kind.sigs.k8s.io/examples/kind-with-registry.sh -o "$TMP_SCRIPT"

# Patch and execute the script
# This adds the ImageVolume feature gate after the apiVersion line
echo "Patching and executing the script with ImageVolume feature gate enabled"
awk '
  /^apiVersion: kind.x-k8s.io\/v1alpha4/ {
    print
    print "featureGates:\n  ImageVolume: true"
    next
  }
  {print}
' "$TMP_SCRIPT" | bash -s

##############################################################################
# 2. Build & push "code-only" OCI image
##############################################################################
WORK=$(mktemp -d)
echo "Workdir: $WORK"
mkdir -p "$WORK/app"

cat > "$WORK/app/index.js" <<'JS'
const http = require('http');
http.createServer((_, res) => res.end('Hello from OCI volume!\n'))
    .listen(process.env.PORT || 80);
JS

cat > "$WORK/Dockerfile" <<'DOCKER'
FROM scratch
COPY app /
DOCKER

RAND_TAG=$(openssl rand -hex 4)
FULL_IMAGE="localhost:5001/hello-app:${RAND_TAG}"

docker build -f "$WORK/Dockerfile" -t "${FULL_IMAGE}" "$WORK"
docker push "${FULL_IMAGE}"

##############################################################################
# 4. Apply Pod manifest (referencing ImageVolume)
##############################################################################
cat > "$WORK/node-demo.yaml" <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: node-demo
spec:
  containers:
  - name: runtime
    image: mirror.gcr.io/node:22-slim
    command: ["node", "/usr/src/app/index.js"]
    workingDir: /usr/src/app
    ports: [{containerPort: 80}]
    volumeMounts:
    - mountPath: /usr/src/app
      name: app-src
  volumes:
  - name: app-src
    image:
      reference: ${FULL_IMAGE}
      pullPolicy: IfNotPresent
EOF

sleep 10
kubectl apply -f "$WORK/node-demo.yaml"
kubectl wait --for=condition=Ready pod/node-demo --timeout=120s

##############################################################################
# 5. Verify it works
##############################################################################
kubectl port-forward pod/node-demo 3000:80 >/dev/null &
PF=$!
sleep 2
echo -n "curl → "
curl http://localhost:3000
kill $PF

cat <<EOM

✅ Done! Node.js is running with code mounted via ImageVolume.

Cleanup:
  kind delete cluster
  rm -rf ${WORK}
EOM

Just copy-paste and run this script to launch a Node.js app using Image Volume in your local environment!

🎯 Use Cases: Where This Shines

Use Case	Traditional Method	With Image Volume
Node.js code distribution (e.g., `node:22.15.1`)	Rebuild with code baked into base	Separate code image, just mount it
Python library sharing (e.g., `python:3.11-slim`)	Build for each app	Fix common env as base, manage apps separately
AI models or certificate deployment	Pack everything into huge images	Share models/certs as Image Volumes

✅ Conclusion: This is the Future Default

"Code-only" images mean faster builds & better cache hits
Base image updates = instant patches
Slimmer CI/CD, improved security

Even if "mounting images" didn't quite click before,
experience firsthand how this mechanism will become the next-generation best practice.

Kubernetes Actions Runner Controller: A Thorough Explanation of PR #4059 Enabling Self-Healing for EphemeralRunner

Kahiro Okina — Thu, 15 May 2025 02:04:38 +0000

1. Background ― Why was this PR necessary?

Symptom	Previous Implementation
Spot instances terminate when their time is up, and EphemeralRunner mistakenly judges this as a failure.	Failures accumulated in EphemeralRunner's `status.failures` one by one.
When using Spot instances with `minRunners ≥ 1`, you inevitably encounter Spot instance interruptions.	After 5 failures (interruptions), the Runner CR remains with `Phase=Failed`.
From the Workflow side, jobs stall in a "Runner not found" state.	Reported about 2 years ago in Issue #2721

2. Previous Limitations and Challenges

The failure limit of 5 times constant existed before
After reaching the limit, it only changed the Phase with markAsFailed(), so:
- EphemeralRunnerSet determined "Desired=1 but object already exists"
- It couldn't generate a new EphemeralRunner and became "zombified"

3. What Changed with PR #4059?

Change	Role
Type change for `status.failures` `map[string]bool` → `map[string]metav1.Time`	Records failure time to track the latest failures chronologically
Introduction of `failedRunnerBackoff` array	Exponential backoff: 0 s → 5 s → 10 s → 20 s → 40 s → 80 s (GitHub)
Added `LastFailure()` helper	Retrieves the most recent failure time from Status (GitHub)
Requeue control	For failure count n ≤ 5, process equivalent to `RequeueAfter(backoff[n])` (GitHub)
Changed handling of limit exceedance	Instead of changing Phase, immediately `Delete()` the EphemeralRunner CR. EphemeralRunnerSet can then regenerate a new one (GitHub)

💡 Point
The constant limit of 5 times itself hasn't changed.
"What happens after reaching the limit" changed from Phase=Failed → object deletion, and
Self-healing of EphemeralRunner became possible

4. Actual Behavior in Spot × minRunners Scenario

sequenceDiagram
    autonumber
    participant ARC as ARC Controller
    participant ER as EphemeralRunner CR
    participant Set as EphemeralRunnerSet
    participant Node as Spot Node

    Node-->>ER: Interruption warning (2 minutes in advance)
    Node--x ER: Pod Terminated
    ARC->>ER: failures+=1 / RequeueAfter(backoff[1])
    loop ~Repeats up to 5 times~
        Node--x ER: Terminated again
        ARC->>ER: failures+=1 / Backoff
    end
    ARC->>ER: failures=6 (>5)\nDelete(ER)
    Set->>ARC: Desired=1 but no CR
    Set-->>ARC: **Create new EphemeralRunner**

On the 6th time, it deletes CR → regenerates, so "zombies" don't remain

5. Code Explanation

1. Status Field for Recording Failures

type EphemeralRunnerStatus struct {
    // ...existing fields
    Failures map[string]metav1.Time `json:"failures,omitempty"`
}

func (s *EphemeralRunnerStatus) LastFailure() *metav1.Time {
    // Helper that scans the map and returns the latest timestamp
}

Key: Pod's .metadata.uid
Value: Time when the failure occurred

2. Exponential Backoff & Requeue

var failedRunnerBackoff = []time.Duration{
    0, 5 * time.Second, 10 * time.Second,
    20 * time.Second, 40 * time.Second, 80 * time.Second,
}
const maxFailures = 5

if len(er.Status.Failures) <= maxFailures {
    delay := failedRunnerBackoff[len(er.Status.Failures)]
    return ctrl.Result{RequeueAfter: delay}, nil
}

Retry with 0 seconds → 5 s → 10 s ...
On the 6th attempt (len > maxFailures), it proceeds to the next section

3. Immediate Deletion on 6th Time → EphemeralRunnerSet Recreates

// Final phase
if len(er.Status.Failures) > maxFailures {
    log.Info("Maximum failures reached – deleting EphemeralRunner")
    if err := r.Delete(ctx, er); err != nil { ... }
    return ctrl.Result{}, nil
}

Since it Deletes the object itself, the Controller's retry ends
The parent resource EphemeralRunnerSet generates a new EphemeralRunner to meet the Desired count → Pod → job resumes

Operation Flow Summary

graph TD
    A[Pod Failed] --> B{Failures <= 5?}
    B -- Yes --> C[Record failure & requeue after backoff]
    B -- No --> D[Delete EphemeralRunner CR]
    D --> E[EphemeralRunnerSet creates new EphemeralRunner]

6. Conclusion

PR #4059 solves the long-standing issue (Issue #2721) of "getting stuck after 5 interruptions" that's particularly frequent in Spot × minRunners environments.

Manages failures chronologically and retries with exponential backoff
Deletes the EphemeralRunner CR upon exceeding the limit, allowing EphemeralRunnerSet to recreate it
This achieves self-healing close to fully managed
However, the risk of job execution being disrupted by Spot interruptions still remains, so caution is needed when using Spot instances

Actions Runner Controller and ArgoCD Integration Guide

Kahiro Okina — Thu, 06 Mar 2025 08:45:58 +0000

Overview

This guide addresses compatibility issues between Actions Runner Controller and ArgoCD, prepared in response to requirements from this PR:
https://github.com/actions/actions-runner-controller/pull/3575#issuecomment-2594906700

1. Create a kind Cluster

kind create cluster --image mirror.gcr.io/kindest/node:v1.32.0

2. Install ArgoCD

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Note: To access the UI, use port forwarding:

kubectl port-forward svc/argocd-server -n argocd 8080:443

3. Create GitHub PAT Secret

Execute the following after the namespace is automatically created via ArgoCD Application syncOptions:

kubectl create secret generic github-pat-secret \
  -n default \
  --from-literal=github_token='<YOUR_PAT>'

Note: Replace <YOUR_PAT> with your actual Personal Access Token.

4. ArgoCD Application: gha-runner-scale-set-controller

Save the following content as gha-runner-scale-set-controller-app.yaml.
This uses Helm Chart version 0.10.1, specifies the image.repository while using the default tag, and adds excludeLabelPropagationPrefixes array settings:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gha-runner-scale-set-controller
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: "ghcr.io/actions/actions-runner-controller-charts"
    chart: "gha-runner-scale-set-controller"
    targetRevision: "0.10.1"
    helm:
      parameters:
        - name: image.repository
          value: "quay.io/kahirokunn/gha-runner-scale-set"
        - name: githubConfigSecret
          value: "github-pat-secret"
        - name: serviceAccount.name
          value: "gha-runner-scale-set-controller"
        - name: flags.excludeLabelPropagationPrefixes[0]
          value: "argocd.argoproj.io/instance"
        - name: flags.excludeLabelPropagationPrefixes[1]
          value: "app.kubernetes.io/component"
        - name: flags.excludeLabelPropagationPrefixes[2]
          value: "app.kubernetes.io/instance"
        - name: flags.excludeLabelPropagationPrefixes[3]
          value: "app.kubernetes.io/managed-by"
        - name: flags.excludeLabelPropagationPrefixes[4]
          value: "app.kubernetes.io/name"
        - name: flags.excludeLabelPropagationPrefixes[5]
          value: "app.kubernetes.io/part-of"
        - name: flags.excludeLabelPropagationPrefixes[6]
          value: "app.kubernetes.io/version"
  destination:
    server: "https://kubernetes.default.svc"
    namespace: arc-systems
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

After applying, it will look like this:

5. ArgoCD Application: demo (Runner Scale Set)

Save the following content as gha-runner-scale-set-app.yaml (filename is arbitrary):
This uses Helm Chart version 0.10.1, sets minimum runners to 1, and configures the following Helm parameters:

controllerServiceAccount.name: "gha-runner-scale-set-controller"
controllerServiceAccount.namespace: "arc-systems"
githubConfigSecret: "github-pat-secret"
githubConfigUrl: "https://github.com/kahirokunn/actions-runner-controller"

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: "ghcr.io/actions/actions-runner-controller-charts"
    chart: "gha-runner-scale-set"
    targetRevision: "0.10.1"
    helm:
      parameters:
        - name: minRunners
          value: "1"
        - name: controllerServiceAccount.name
          value: "gha-runner-scale-set-controller"
        - name: controllerServiceAccount.namespace
          value: "arc-systems"
        - name: githubConfigSecret
          value: "github-pat-secret"
        - name: githubConfigUrl
          value: "https://github.com/kahirokunn/actions-runner-controller"
  destination:
    server: "https://kubernetes.default.svc"
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

After applying, it will appear in ArgoCD as follows: