DEV Community

Kahiro Okina
Kahiro Okina

Posted on

Introduction to KEP-5339: Plugin for Credentials in ClusterProfile

#kubernetes #aws #eks #go

Introduction
This article summarizes the key points of KEP-5339 "Plugin for Credentials in ClusterProfile" (SIG-Multicluster) and presents a simple implementation example using EKS.
KEP: https://github.com/kubernetes/enhancements/tree/master/keps/sig-multicluster/5339-clusterprofile-plugin-credentials

Project Introduction: ClusterProfile Credentials Plugins

To make ClusterProfile Credentials Plugins easily testable, I've released a repository containing ready-to-use plugins.

  • Repository: labthrust/clusterprofile-credentials-plugins

  • Included plugins (all implemented as single binaries in Go):

    • eks-aws-auth-plugin: Resolves clusters from EKS endpoints and CA, returns ExecCredential
    • secretreader-plugin: Reads data.token from Kubernetes Secrets, returns ExecCredential

This repository aims to become an Awesome Plugins style catalog. I plan to continue adding and introducing useful plugins, with SPIRE plugin being one consideration for the future.

The reason I started this project is that while Credentials Plugin is an excellent specification, there's currently a significant hurdle of "having to create your own plugin first." Additionally, information about which plugins are available, how to use them, and where to find the binaries is not well organized.
Therefore, I've released this as a "collection of plugins you can immediately deploy and try" to lower the barrier for taking the first steps with ClusterProfile.

Overview

  • ClusterProfile's status.credentialProviders[].cluster holds server / CA.
  • The controller registers exec plugins in cp-creds.json and obtains tokens using KUBERNETES_EXEC_INFO passed at runtime as input.
  • Using .spec.cluster.server as the starting point, it identifies EKS clusters and calls aws eks get-token.
  • Input and output are in ExecCredential format.

Overall Flow (Mermaid) 


sequenceDiagram
  autonumber
  participant Ctrl as Controller
  participant Prov as Credential Provider (cp-creds.json)
  participant Plugin as exec plugin<br/>(./eks-aws-auth-plugin.sh)
  participant AWS as AWS EKS API

  Ctrl->>Prov: Load providers
  Ctrl->>Plugin: exec providers[].execConfig.command<br/>env: KUBERNETES_EXEC_INFO
  Note right of Plugin: Extract region from server<br/>List EKS → resolve clusterName by exact endpoint match

  Plugin->>AWS: eks list-clusters (region)
  Plugin->>AWS: eks describe-cluster (each cluster's endpoint)
  AWS-->>Plugin: endpoint list
  Plugin->>AWS: eks get-token (clusterName)
  AWS-->>Plugin: ExecCredential(JSON)<br/>status.token, expirationTimestamp

  Plugin-->>Ctrl: Return ExecCredential(JSON) via stdout
Enter fullscreen mode Exit fullscreen mode

Hands-On Example

Complete sample: https://github.com/kahirokunn/cluster-inventory-api/tree/eks-example

1) Start Controller 

go build  controller_example.go
./controller_example -clusterprofile-provider-file ./cp-creds.json
Enter fullscreen mode Exit fullscreen mode

2) Contents of cp-creds.json

This is the contents of the JSON file passed with the -clusterprofile-provider-file flag:

{
  "providers": [
    {
      "name": "eks",
      "execConfig": {
          "apiVersion": "client.authentication.k8s.io/v1beta1",
          "args": null,
          "command": "./eks-aws-auth-plugin.sh",
          "env": null,
          "provideClusterInfo": true
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

3) ClusterProfile YAML State 

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ClusterProfile
metadata:
  name: my-cluster-1
spec:
  displayName: my-cluster-1
  clusterManager:
    name: EKS-Fleet
status:
  credentialProviders:
  - name: eks
    cluster:
      server: https://xxx.gr7.ap-northeast-1.eks.amazonaws.com
      certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1J...
Enter fullscreen mode Exit fullscreen mode

ExecCredential and exec plugin Flow

  • ExecCredential
    Kubernetes exec authentication plugin protocol. At runtime, input is provided via the KUBERNETES_EXEC_INFO environment variable, and output returns ExecCredential to stdout.
    Specification: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#input-and-output-formats

  • Usage with kubectl / client-go
    When you configure a command in kubeconfig's users[].exec or in credentials provider's providers[].execConfig as in this article, the command is launched during authentication. KUBERNETES_EXEC_INFO is passed to the plugin at that time, and kubectl/client-go consumes the ExecCredential from the plugin's stdout.

  • Current Configuration
    ./eks-aws-auth-plugin.sh is specified in providers[].execConfig.command. The plugin uses .spec.cluster.server and CA from KUBERNETES_EXEC_INFO as clues to identify the cluster and ultimately outputs ExecCredential.

Background: "Compatibility" of aws eks update-kubeconfig / aws eks get-token

  • aws eks update-kubeconfig

    • Function: Command that generates/updates local kubeconfig.
    • Input: Requires explicit cluster name specification via --name, etc. No mechanism to receive KUBERNETES_EXEC_INFO (ExecCredential's spec.cluster.server / CA).
    • Output: Does not return ExecCredential JSON to stdout. Only modifies kubeconfig files. → Conclusion: Not compatible with exec plugin input/output requirements of "input = KUBERNETES_EXEC_INFO, output = ExecCredential(JSON)/stdout".

  • aws eks get-token

    • Output: Returns ExecCredential-compatible JSON to stdout.
    • Input: Requires explicit specification of --cluster-name. Cannot directly receive KUBERNETES_EXEC_INFO as input (lacks functionality to resolve cluster names from just server/CA). → Conclusion: Output is compatible, but input is not. A separate resolution layer from KUBERNETES_EXEC_INFO (server/CA) → cluster-name is needed.

EKS exec plugin Implementation

Processing flow:

  1. Get .spec.cluster.server from KUBERNETES_EXEC_INFO
  2. Extract region from hostname
  3. List EKS in that region and identify cluster name by exact endpoint match
  4. Execute aws eks get-token 
#!/usr/bin/env bash
set -euo pipefail

# -------- utils --------
err() { printf "[eks-exec-credential] %s\n" "$*" >&2; }
need() { command -v "$1" >/dev/null 2>&1 || { err "missing dependency: $1"; exit 1; }; }
normalize_host() { sed -E 's#^https?://##; s#/$##; s#:443$##'; }

need jq
need aws

# --- read ExecCredential ---
if [[ -z "${KUBERNETES_EXEC_INFO:-}" ]]; then
  err "KUBERNETES_EXEC_INFO is empty. set provideClusterInfo: true"
  exit 1
fi

REQ_API_VERSION="$(jq -r '.apiVersion // empty' <<<"$KUBERNETES_EXEC_INFO")"
SERVER="$(jq -r '.spec.cluster.server // empty' <<<"$KUBERNETES_EXEC_INFO")"
if [[ -z "$SERVER" || "$SERVER" == "null" ]]; then
  err "spec.cluster.server is missing in KUBERNETES_EXEC_INFO"
  exit 1
fi

NORM_SERVER="$(printf "%s" "$SERVER" | normalize_host)"

# --- region: infer from server hostname ---
HOST="${NORM_SERVER%%/*}"
REGION="$(printf "%s\n" "$HOST" \
  | sed -nE 's#.*\.([a-z0-9-]+)\.eks(-fips)?\.amazonaws\.com(\.cn)?$#\1#p')"
if [[ -z "$REGION" ]]; then
  err "failed to parse region from server hostname: ${SERVER}"
  err "expected something like ...<random>.<suffix>.<region>.eks.amazonaws.com"
  exit 1
fi

# --- tiny cache: endpoint -> cluster name ---
CACHE_DIR="${XDG_CACHE_HOME:-$HOME/.cache}/eks-exec-credential"
mkdir -p "$CACHE_DIR"
MAP_CACHE="$CACHE_DIR/endpoint-map-${REGION}.json"
if [[ ! -s "$MAP_CACHE" ]] || ! jq -e . >/dev/null 2>&1 <"$MAP_CACHE"; then
  echo '{}' >"$MAP_CACHE"
fi

lookup_cache() { jq -r --arg k "$NORM_SERVER" '.[$k] // empty' <"$MAP_CACHE"; }
update_cache() {
  local tmp; tmp="$(mktemp)"
  jq --arg k "$NORM_SERVER" --arg v "$1" '.[$k]=$v' "$MAP_CACHE" >"$tmp" && mv "$tmp" "$MAP_CACHE"
}

match_endpoint() {
  local name="$1"
  local ep norm_ep
  ep="$(aws eks describe-cluster --region "$REGION" --name "$name" \
        --query 'cluster.endpoint' --output text 2>/dev/null || true)"
  [[ -z "$ep" || "$ep" == "None" ]] && return 1
  norm_ep="$(printf "%s" "$ep" | normalize_host)"
  [[ "$norm_ep" == "$NORM_SERVER" ]]
}

CLUSTER_NAME=""
# 1) cache hit?
CACHED="$(lookup_cache || true)"
if [[ -n "$CACHED" ]] && match_endpoint "$CACHED"; then
  CLUSTER_NAME="$CACHED"
fi

# 2) enumerate if needed
if [[ -z "$CLUSTER_NAME" ]]; then
  err "resolving cluster in ${REGION} for ${NORM_SERVER}"
  found=""
  while IFS= read -r name; do
    [[ -z "$name" ]] && continue
    if match_endpoint "$name"; then
      found="$name"
      break
    fi
  done < <(aws eks list-clusters --region "$REGION" --output json | jq -r '.clusters[]?')

  if [[ -z "$found" ]]; then
    err "no matching EKS cluster for endpoint: ${SERVER} (region=${REGION})"
    exit 1
  fi
  CLUSTER_NAME="$found"
  update_cache "$CLUSTER_NAME" || true
fi

# --- fetch ExecCredential via aws CLI ---
TOKEN_JSON="$(aws eks get-token --region "$REGION" --cluster-name "$CLUSTER_NAME" --output json)"

printf "%s\n" "$TOKEN_JSON"
Enter fullscreen mode Exit fullscreen mode

Controller Usage Example

Sample code that generates rest.Config and creates a Clientset.

package main

import (
    "context"
    "encoding/base64"
    "flag"
    "log"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    clientcmdv1 "k8s.io/client-go/tools/clientcmd/api/v1"
    "sigs.k8s.io/cluster-inventory-api/apis/v1alpha1"
    "sigs.k8s.io/cluster-inventory-api/pkg/credentials"
)

func main() {
    credentialsProviders := credentials.SetupProviderFileFlag()
    flag.Parse()

    cpCreds, err := credentials.NewFromFile(*credentialsProviders)
    if err != nil {
        log.Fatalf("Got error reading credentials providers: %v", err)
    }

    caPEMBase64 := `LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1J...`
    caPEM, err := base64.StdEncoding.DecodeString(caPEMBase64)
    if err != nil {
        log.Fatalf("CA PEM base64 decode failed: %v", err)
    }

    // normally we would get this clusterprofile from the local cluster (maybe a watch?)
    // and we would maintain the restconfigs for clusters we're interested in.
    exampleClusterProfile := v1alpha1.ClusterProfile{
        Spec: v1alpha1.ClusterProfileSpec{
            DisplayName: "My Cluster",
        },
        Status: v1alpha1.ClusterProfileStatus{
            CredentialProviders: []v1alpha1.CredentialProvider{
                {
                    Name: "eks",
                    Cluster: clientcmdv1.Cluster{
                        Server: "https://xxx.gr7.ap-northeast-1.eks.amazonaws.com",
                        CertificateAuthorityData: caPEM,
                    },
                },
            },
        },
    }

    restConfigForMyCluster, err := cpCreds.BuildConfigFromCP(&exampleClusterProfile)
    if err != nil {
        log.Fatalf("Got error generating restConfig: %v", err)
    }
    log.Printf("Got credentials: %v", restConfigForMyCluster)
    // I can then use this rest.Config to build a k8s client.

    // Build a client and list Pods in the default namespace
    clientset, err := kubernetes.NewForConfig(restConfigForMyCluster)
    if err != nil {
        log.Fatalf("failed to create clientset: %v", err)
    }
    ctx := context.Background()
    pods, err := clientset.CoreV1().Pods("default").List(ctx, metav1.ListOptions{})
    if err != nil {
        log.Fatalf("failed to list pods: %v", err)
    }
    log.Printf("default namespace has %d pods", len(pods.Items))
    for i, p := range pods.Items {
        if i >= 10 {
            log.Printf("... (truncated)")
            break
        }
        log.Printf("pod: %s", p.Name)
    }
}
Enter fullscreen mode Exit fullscreen mode

Troubleshooting

  • no matching EKS cluster → Check results, permissions, and profiles for aws eks list-clusters --region <r>
  • x509: certificate signed by unknown authority → Verify certificate-authority-data (base64)

Additional Notes

ExecCredential allows passing additional values using extensions.

KEP-5339: ClusterProfile's Credentials Plugin does not define specifications for using ExecCredential.extensions.
Related code: https://github.com/kubernetes-sigs/cluster-inventory-api/blob/main/pkg/credentials/config.go#L133

References

Top comments (0)