DEV Community

Cover image for Multi-Cluster com Karmada [Lab Session]
AWS Community Builders profile image Paulo Ponciano
Paulo Ponciano for AWS Community Builders

Posted on • Edited on

Multi-Cluster com Karmada [Lab Session]

#kubernetes #karmada #community

Capa do lab Multi-Cluster com Karmada

What is Karmada?

Karmada (Kubernetes Armada) is a Kubernetes management system that enables you to run your cloud-native applications across multiple Kubernetes clusters and clouds, with no changes to your applications. By speaking Kubernetes-native APIs and providing advanced scheduling capabilities, Karmada enables truly open, multi-cloud Kubernetes.

Karmada aims to provide turnkey automation for multi-cluster application management in multi-cloud and hybrid cloud scenarios, with key features such as centralized multi-cloud management, high availability, failure recovery, and traffic scheduling.

Architecture

Diagrama da arquitetura do Karmada

Source: https://karmada.io

Arquitetura do Lab

Arquitetura do lab multi-cluster com Karmada nos clusters pegasus e pegasus-2

Deploy da infra

  • Cluster 1 ou pegasus: 
git clone https://github.com/paulofponciano/EKS-Istio-Karpenter-ArgoCD.git
Enter fullscreen mode Exit fullscreen mode

Altere os valores em variables.tfvars se for necessário.
No arquivo nlb.tf, estamos informando um certificado do ACM. Altere para o seu certificado ou remova comentando as linhas 38, 39, 40 e descomentando a linha 37.

Com certificado:

resource "aws_lb_listener" "ingress_443" {
  load_balancer_arn = aws_lb.istio_ingress.arn
  port              = "443"
  #protocol          = "TCP"
  protocol        = "TLS"
  certificate_arn = "arn:aws:acm:us-east-2:ACCOUNTID:certificate/bfbfe3ce-d347-4c42-8986-f45e95e04ca1"
  alpn_policy     = "HTTP2Preferred"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.https.arn
  }
}
Enter fullscreen mode Exit fullscreen mode

Sem certificado:

resource "aws_lb_listener" "ingress_443" {
  load_balancer_arn = aws_lb.istio_ingress.arn
  port              = "443"
  protocol          = "TCP"
  # protocol        = "TLS"
  # certificate_arn = "arn:aws:acm:us-east-2:ACCOUNTID:certificate/bfbfe3ce-d347-4c42-8986-f45e95e04ca1"
  # alpn_policy     = "HTTP2Preferred"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.https.arn
  }
}
Enter fullscreen mode Exit fullscreen mode 
tofu init
tofu plan --var-file variables.tfvars
tofu apply --var-file variables.tfvars
Enter fullscreen mode Exit fullscreen mode
  • Cluster 2 ou pegasus-2: 
git clone https://github.com/paulofponciano/EKS-Istio-Karpenter.git
Enter fullscreen mode Exit fullscreen mode

Altere os valores em variables.tfvars se for necessário.

tofu init
tofu plan --var-file variables.tfvars
tofu apply --var-file variables.tfvars
Enter fullscreen mode Exit fullscreen mode

Deploy do karmada 

git clone https://github.com/paulofponciano/karmada.git
cd karmada
Enter fullscreen mode Exit fullscreen mode

Entrando no contexto do cluster pegasus:

aws eks update-kubeconfig --region us-east-2 --name pegasus
Enter fullscreen mode Exit fullscreen mode

Atualização do kubeconfig para o contexto do cluster pegasus

Os comandos a seguir serão executados no cluster pegasus (Cluster 1), onde temos o Argo rodando.

helm repo add karmada-charts https://raw.githubusercontent.com/karmada-io/karmada/master/charts
helm repo update
helm --namespace karmada-system upgrade -i karmada karmada-charts/karmada --create-namespace
Enter fullscreen mode Exit fullscreen mode

Instalação do Karmada via Helm no namespace karmada-system

Verificando o deployment:

kubectl get pods -n karmada-system
Enter fullscreen mode Exit fullscreen mode

Pods do controlplane do Karmada em execução

Agora já temos o controlplane do karmada rodando no cluster pegasus.

Com o deploy, é gerado um secret com o kubeconfig necessário para conectarmos no controlplane do karmada:

kubectl get secrets -n karmada-system | grep karmada-kubeconfig
Enter fullscreen mode Exit fullscreen mode

Secret karmada-kubeconfig listado no namespace karmada-system

kubectl get secret -n karmada-system karmada-kubeconfig -o jsonpath='{.data.kubeconfig}' | base64 --decode
Enter fullscreen mode Exit fullscreen mode

Criando IRSA (Iam Role for Service Account)

Altere os valores de ACCOUNTID na iam-policy-irsa-karmada.json e no comando eksctl abaixo.

Isso refletirá em uma Service Account dentro do cluster pegasus, usaremos ela montando em um pod do ubuntu como ponto de acesso temporário ao controlplane do karmada.

aws iam create-policy --policy-name ubuntu-admin-karmada \
  --policy-document file://iam-policy-irsa-karmada.json
Enter fullscreen mode Exit fullscreen mode 
eksctl create iamserviceaccount --name ubuntu-admin-karmada \
  --namespace karmada-system \
  --cluster pegasus \
  --attach-policy-arn arn:aws:iam::ACCOUNTID:policy/ubuntu-admin-karmada \
  --region us-east-2 \
  --profile default \
  --approve
Enter fullscreen mode Exit fullscreen mode

Criação da IRSA ubuntu-admin-karmada via eksctl

No lado AWS, isso reflete em uma IAM role que podemos adicionar nos dois clusters EKS.

  • EKS IAM - Console (pegasus-2):

IAM role adicionada no console do EKS para o cluster pegasus-2

Configuração de acesso da IAM role no cluster pegasus-2

Faça o mesmo para o cluster pegasus, onde o karmada controlplane está rodando.

Acessando Karmada API-Server

Vamos subir agora aquele pod do ubuntu-admin no cluster pegasus. No manifesto já está tudo definido para utilizar a Service Account que criamos mais acima.

kubectl apply -f https://raw.githubusercontent.com/paulofponciano/karmada/main/ubuntu-admin-karmada.yaml
kubectl get pods -n karmada-system | grep ubuntu
Enter fullscreen mode Exit fullscreen mode

Pod ubuntu-admin-karmada em execução no namespace karmada-system

Nesse momento, vamos entrar no container do ubuntu, que está rodando no cluster pegasus:

kubectl exec -it ubuntu-admin-karmada -n karmada-system -- /bin/bash
Enter fullscreen mode Exit fullscreen mode
  • Instalando o kubectl karmada: 
curl -s https://raw.githubusercontent.com/karmada-io/karmada/master/hack/install-cli.sh | bash -s kubectl-karmada
Enter fullscreen mode Exit fullscreen mode

Instalação do plugin kubectl karmada dentro do container

  • Entrando no contexto do cluster pegasus-2: 
aws eks update-kubeconfig --region us-west-2 --name pegasus-2 --kubeconfig $HOME/.kube/pegasus-2.config
kubectl get nodes --kubeconfig $HOME/.kube/pegasus-2.config
Enter fullscreen mode Exit fullscreen mode

Nodes do cluster pegasus-2 listados via kubeconfig dedicado

  • Vamos fazer o mesmo para o cluster pegasus: 
aws eks update-kubeconfig --region us-east-2 --name pegasus --kubeconfig $HOME/.kube/pegasus.config
kubectl get nodes --kubeconfig $HOME/.kube/pegasus.config
Enter fullscreen mode Exit fullscreen mode

Nodes do cluster pegasus listados via kubeconfig dedicado

  • Checando acesso ao karmada apiserver: 
kubectl get all -A --kubeconfig /etc/karmada-kubeconfig/kubeconfig
Enter fullscreen mode Exit fullscreen mode

Recursos do Karmada apiserver listados com kubectl get all

  • Join do pegasus-2 no karmada: 
kubectl karmada --kubeconfig /etc/karmada-kubeconfig/kubeconfig join pegasus-2 --cluster-kubeconfig=$HOME/.kube/pegasus-2.config
Enter fullscreen mode Exit fullscreen mode

Join do cluster pegasus-2 no Karmada

  • Join do pegasus no karmada: 
kubectl karmada --kubeconfig /etc/karmada-kubeconfig/kubeconfig join pegasus --cluster-kubeconfig=$HOME/.kube/pegasus.config
Enter fullscreen mode Exit fullscreen mode

Join do cluster pegasus no Karmada

  • Checando status dos clusters adicionados: 
kubectl --kubeconfig /etc/karmada-kubeconfig/kubeconfig get clusters
Enter fullscreen mode Exit fullscreen mode

Clusters pegasus e pegasus-2 com status Ready no Karmada

  • Instalando CLI do ArgoCD: 
curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
install -m 555 argocd-linux-amd64 /usr/local/bin/argocd
rm argocd-linux-amd64
Enter fullscreen mode Exit fullscreen mode
  • Recuperando o secret / password do Argo server e fazendo login: 
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
Enter fullscreen mode Exit fullscreen mode 
argocd login argocd-server.argocd.svc.cluster.local:80 --username admin
Enter fullscreen mode Exit fullscreen mode

Login na CLI do ArgoCD a partir do container

  • Adicionando o karmada como cluster no ArgoCD: 
argocd cluster add karmada-apiserver --kubeconfig /etc/karmada-kubeconfig/kubeconfig --name karmada-controlplane
Enter fullscreen mode Exit fullscreen mode

Karmada adicionado como cluster no ArgoCD

Deploy com ArgoCD

Se acessarmos a UI do Argo, que está rodando no cluster pegasus, veremos que o karmada está registrado como um cluster onde é possível o Argo fazer deploy:

Karmada registrado como cluster de destino na UI do ArgoCD

Podemos agora aplicar um manifesto que irá definir uma nova fonte para o Argo buscar por deploys. Nesse caso, essa fonte é um repositório no GitHub:

kubectl apply -f karmada-argo-app.yaml
Enter fullscreen mode Exit fullscreen mode

Como já temos manifestos nesse repositório (no path /app-manifests), o Argo já faz o sync entregando essas aplicações no controlplane do karmada e o karmada, por sua vez, entrega nos dois clusters de acordo com o que for definido nos manifestos de PropagationPolicy:

Aplicações sincronizadas pelo Argo no controlplane do Karmada

Detalhe das aplicações distribuídas pelo Karmada nos clusters

No cluster pegasus podemos ver:

kubectl get pods -o wide | grep redis
Enter fullscreen mode Exit fullscreen mode

Pods do redis em execução no cluster pegasus

Cluster pegasus-2:

kubectl get pods -o wide | grep nginx
Enter fullscreen mode Exit fullscreen mode

Pods do nginx em execução no cluster pegasus-2

No caso do deploy do RabbitMQ, podemos ver que existem réplicas rodando nos dois clusters; quando olharmos os arquivos de PropagationPolicy poderemos entender.

kubectl get pods -o wide --context arn:aws:eks:us-east-2:ACCOUNTID:cluster/pegasus | grep rabbitmq
Enter fullscreen mode Exit fullscreen mode

Pods do RabbitMQ no cluster pegasus

kubectl get pods -o wide --context arn:aws:eks:us-west-2:ACCOUNTID:cluster/pegasus-2 | grep rabbitmq
Enter fullscreen mode Exit fullscreen mode

Pods do RabbitMQ no cluster pegasus-2

Karmada OverridePolicy e PropagationPolicy

No repositório que o Argo está monitorando, podemos ver os manifestos do karmada e também os manifestos de deployment que usamos como exemplo.

Exemplo Redis

Regras de override e selector do deployment onde será aplicado, no caso 'redis':

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: redis-op
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: redis
  overrideRules:
    - targetCluster:
        clusterNames:
          - pegasus-2
      overriders:
        labelsOverrider:
          - operator: add
            value:
              env: skoala-dev
          - operator: add
            value:
              env-stat: skoala-stage
          - operator: remove
            value:
              for: for
          - operator: replace
            value:
              bar: test
    - targetCluster:
        clusterNames:
          - pegasus
      overriders:
        annotationsOverrider:
          - operator: add
            value:
              env: skoala-stage
          - operator: remove
            value:
              bom: bom
          - operator: replace
            value:
              emma: sophia
Enter fullscreen mode Exit fullscreen mode

Regras de propagação, selector do deployment e target cluster. Nesse caso de failover, esse deployment deve ser migrado para o cluster pegasus-2 caso o cluster pegasus entre em falha:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: redis-propagation
spec:
  propagateDeps: true
  failover:
    application:
      decisionConditions:
        tolerationSeconds: 120
      purgeMode: Never
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: redis
  placement:
    clusterAffinity:
      clusterNames:
        - pegasus
        - pegasus-2
    spreadConstraints:
      - maxGroups: 1
        minGroups: 1
        spreadByField: cluster
Enter fullscreen mode Exit fullscreen mode

Exemplo Nginx 

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: nginx-op
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  overrideRules:
    - targetCluster:
        clusterNames:
          - pegasus-2
      overriders:
        labelsOverrider:
          - operator: add
            value:
              env: skoala-dev
          - operator: add
            value:
              env-stat: skoala-stage
          - operator: remove
            value:
              for: for
          - operator: replace
            value:
              bar: test
Enter fullscreen mode Exit fullscreen mode

Neste caso, apenas o cluster pegasus-2 foi definido em 'targetCluster':

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  placement:
    clusterAffinity:
      clusterNames:
        - pegasus-2
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
          - targetCluster:
              clusterNames:
                - pegasus-2
            weight: 1
Enter fullscreen mode Exit fullscreen mode

Exemplo RabbitMQ 

apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
  name: rabbitmq-op
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: rabbitmq
  overrideRules:
    - targetCluster:
        clusterNames:
          - pegasus-2
      overriders:
        labelsOverrider:
          - operator: add
            value:
              env: skoala-dev
          - operator: add
            value:
              env-stat: skoala-stage
          - operator: remove
            value:
              for: for
          - operator: replace
            value:
              bar: test
    - targetCluster:
        clusterNames:
          - pegasus
      overriders:
        annotationsOverrider:
          - operator: add
            value:
              env: skoala-stage
          - operator: remove
            value:
              bom: bom
          - operator: replace
            value:
              emma: sophia
Enter fullscreen mode Exit fullscreen mode

Aqui temos algo diferente, onde os dois clusters são definidos em 'targetCluster', porém com pesos (weights) diferentes, fazendo com que o karmada entregue as réplicas de acordo com o peso de cada cluster:

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: rabbitmq-propagation
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: rabbitmq
  placement:
    clusterAffinity:
      clusterNames:
        - pegasus
        - pegasus-2
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
          - targetCluster:
              clusterNames:
                - pegasus
            weight: 2
          - targetCluster:
              clusterNames:
                - pegasus-2
            weight: 1
Enter fullscreen mode Exit fullscreen mode

Remover ubuntu-admin e IRSA

Podemos deletar o pod ubuntu que usamos para setup do karmada, e também a IRSA:

kubectl delete -f ubuntu-admin-karmada.yaml
Enter fullscreen mode Exit fullscreen mode 
eksctl delete iamserviceaccount --name ubuntu-admin-karmada \
  --namespace karmada-system \
  --cluster pegasus \
  --region us-east-2 \
  --profile default
Enter fullscreen mode Exit fullscreen mode

Keep shipping!

Top comments (0)