Setup Rancher on EKS + ALB

#eks #rancher #aws #kubernetes

In this article, I am going to show you how to setup Rancher on EKS with Application Load Balancer (ALB).

In case of using Amazon Certificate Manager (ACM) and want to terminate the SSL certificate on the Load Balancer:

Classic Load Balancer: Rancher needs to use WebSocket and Classic Load Balancer is not supporting WebSocket also AWS is not recommended to use it.
Network Load Balancer (NLB): is terminating traffic in tcp mode for port 443 rather than tls mode.
The NLB does not inject the correct headers into requests when terminated at the NLB.
Application Load Balancer (ALB):
if you want to use certificates managed by ACM, you should use ALB.

Rancher documentation is using nginx-ingress-controller and only creates Classic Load Balancer or Network Load Balancer.
We will use AWS Load Balancer Controller to create ALB for our Rancher.

Steps:

1- Create EKS Cluster
2- Install AWS Load Balancer Controller
3- Register a domain in route53 or create a subdomain, ex: rancher.example.com
4- Request a certificate from ACM
5- Install Rancher:

Add the Helm Chart Repository

helm repo add rancher-stable https://releases.rancher.com/server-charts/stable

In the command below add the following:

The domain.
The ARN of your SSL certificate in step 3.
The public subnets ids of your VPC that will be used for the ALB.

helm install rancher rancher-stable/rancher --namespace cattle-system --set hostname=rancher.example.com --set 'ingress.extraAnnotations.alb\.ingress\.kubernetes\.io/scheme=internet-facing' --set 'ingress.extraAnnotations.alb\.ingress\.kubernetes\.io/success-codes=200\,404\,301\,302' --set 'ingress.extraAnnotations.alb\.ingress\.kubernetes\.io/subnets=subnet-XXX\,subnet-XXX\,subnet-XXX' --set 'ingress.extraAnnotations.alb\.ingress\.kubernetes\.io/listen-ports=[{\"HTTP\": 80}\, {\"HTTPS\": 443}]' --set 'ingress.extraAnnotations.alb\.ingress\.kubernetes\.io/certificate-arn=arn:aws:acm:eu-central-1:XXX:certificate/XXX' --set 'ingress.extraAnnotations.kubernetes\.io/ingress\.class=alb'  --set replicas=3 --set tls=external --create-namespace

6- Change the rancher service to use NodePort rather than ClusterIP, AWS Load Balancer Controller is not using ClusterIP

kubectl -n cattle-system patch svc rancher -p '{"spec": {"type": "NodePort"}}'

7- Update your domain in route53 to point to the Application Load Balancer

8- Create the password first time for the admin user

echo https://rancher.example.com/dashboard/?setup=$(kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}')

9- Create Nginx Ingress Controller
Kubectl exec/port-forward with AWS ALB and nginx-ingress-controller

10- To Enable auditing
By default audit is disabled, to enable Audit, you need to update the values
Get the existing values

helm get values  rancher --namespace cattle-system > values.yaml

Update the values.yaml and looks like the following:

auditLog:
  level: 2
hostname: rancher.example.com
ingress:
  extraAnnotations:
    alb.ingress.kubernetes.io/conditions.rancher: >
      [{"field":"path-pattern","pathPatternConfig":{"values":["/*"]}}]
    alb.ingress.kubernetes.io/certificate-arn: XXXXX
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/subnets: subnet-XXX,subnet-XXX,subnet-XXX
    alb.ingress.kubernetes.io/success-codes: 200,404,301,302
    kubernetes.io/ingress.class: alb
replicas: 3
tls: external

Update the values with the Rancher same version

helm upgrade rancher rancher-stable/rancher --namespace cattle-system --version=v2.X.X --values values.yaml

To get the audit logs

kubectl logs -n cattle-system -l app=rancher -c rancher-audit-log

To upgrade Rancher

helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
helm repo update
helm search repo rancher-stable
helm upgrade rancher rancher-stable/rancher --namespace cattle-system
helm history rancher --namespace cattle-system

Sources:
https://docs.aws.amazon.com/
https://rancher.com/docs/
https://images.app.goo.gl/JiWMfcZoZJGQWRGy9

Top comments (5)

Laurie Kepford • May 4 '22

I have been looking everywhere for something like this! I tried to use this but got this error:
Error: UPGRADE FAILED: failed to create resource: Internal error occurred: failed calling webhook "vingress.elbv2.k8s.aws": Post "aws-load-balancer-webhook-service.... no endpoints available for service "aws-load-balancer-webhook-service"

Notice the webhook name starts with a v.

No ALB is created.

Laurie Kepford • May 4 '22

I think my cluster was broken, because I started from scratch it worked like a charm.

Don Laidlaw • Nov 24 '22

Why use nginx at all? Just set the ALB to ip targeting mode and let the ALB route to the rancher service. The controller will keep the ALB target-group up-to-date as the service endpoints change.

For example: use a values file like:

hostname: rancher.yourdomain.com
replicas: 3
tls: external
restrictedAdmin: true
ingress:
  ingressClassName: alb
  extraAnnotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/success-codes: '200,404,301,302'
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:00000012345:certificate/lots-of-hex-chars
    alb.ingress.kubernetes.io/security-groups: sg-000000001234
    alb.ingress.kubernetes.io/load-balancer-name: rancher
    alb.ingress.kubernetes.io/tags: 'Environment=dev,Mode=test'
    alb.ingress.kubernetes.io/backend-protocol: HTTP
    external-dns.alpha.kubernetes.io/hostname: rancher.yourdomain.com

Then the install command is like:

helm upgrade --install rancher rancher-stable/rancher \
--namespace cattle-system \
--create-namespace \
--values local-values.yaml \
--set bootstrapPassword='YourSecurePassword'

No need for an nginx ingress controller here.