Kubectl exec/port-forward with AWS ALB and nginx-ingress-controller

I was facing issues when I performed kubectl with exec or port-forward option on my Rancher clusters that used EKS and ALB, it was giving me this error

kubectl exec -it app -- bash

Error from server (BadRequest): Upgrade request required

exec and port forward are using SPDY protocol and ALB does not support it.

The HTTPS request is going from the user to ALB, then SSL is terminated on the ALB, and the request is forwarded to the Nginx controller service after that forward to the rancher service.

See the part 1 to setup Rancher on EKS and ALB

After that, you need to do the following:

1- Install Nginx Ingress Controller

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

Kubernetes v1.16+ use version 3.x.x

helm upgrade --install nginx ingress-nginx/ingress-nginx --namespace cattle-system --set controller.service.type=NodePort  --set controller.service.targetPorts.https=http --set-string controller.config.use-forwarded-headers="true" --version 3.12.0

Kubernetes v1.19+ use version 4.x.x

helm upgrade --install nginx ingress-nginx/ingress-nginx --namespace cattle-system --set controller.service.type=NodePort  --set controller.service.targetPorts.https=http --set-string controller.config.use-forwarded-headers="true" --version 4.2.3

2- Edit the rancher Ingress

kubectl edit ingress -n cattle-system rancher

Change the host and name inside spec with the following:

spec:
   rules:
   - host: '*.example.com'
     http:
       paths:
       - backend:
           service:
             name: nginx-ingress-nginx-controller
             port:
               number: 80
         pathType: ImplementationSpecific

3- Create a new ingress with the following:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
  name: rancher-exec
  namespace: cattle-system

spec:
   rules:
   - host: 'rancher.example.com'
     http:
       paths:
       - backend:
           service:
             name: rancher
             port:
               number: 80
         pathType: ImplementationSpecific

You need to add ingressClassName: nginx only for rancher-exec

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
  name: rancher-exec
  namespace: cattle-system

spec:
   ingressClassName: nginx 
   rules:
   - host: 'rancher.example.com'
     http:
       paths:
       - backend:
           service:
             name: rancher
             port:
               number: 80
         pathType: ImplementationSpecific

Note: It will remove the ALB if you add ingressClassName: nginx to rancher ingress

Test the kubectl exec and port-forward

kubectl create deployment nginx2 --image nginx:alpine
kubectl expose deployment nginx2 --port=80
kubectl exec -it nginx2-XXXXX -- sh
kubectl port-forward service/nginx2 --address 0.0.0.0 80:80

Comments

[Rez0k]

Great article!

I encountered another problem after implementing your solution, the terminal session in Rancher was closed after about 60 seconds.

The reason was that the nginx controller config was set to the default of 60 seconds.
to solve this add to the nginx helm command:

--set-string controller.config.proxy-connect-timeout: "1800" --set-string controller.config.proxy-read-timeout="1800"

This will increase the terminal session to half an hour (1800 seconds).