Creating Network Load Balancer (SG supported) with AWS Load Balancer Controller

AWS Network Load Balancer (NLB) has finally added support for Security Groups.
https://aws.amazon.com/about-aws/whats-new/2023/08/network-load-balancer-supports-security-groups/

This significant update means that AWS Load Balancer Controller now automatically attaches Security Groups to NLBs by default.

What Has Changed

Detailed behavior changes can be found in the release notes for version 2.6.0 of the AWS Load Balancer Controller.
https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases
The key points include:

• Similar to Application Load Balancers (ALB) in Ingress, if you do not explicitly specify the attached Security Group, two Security Groups are automatically created and attached:
  • One for receiving external traffic (frontend SG)
  • One for communicating with the backend Node Groups (backend SG), which is automatically added to the Node Group's Security Group ingress rules.
• You can explicitly specify the frontend SG using the service.beta.kubernetes.io/aws-load-balancer-security-groups annotation.
• Existing NLBs without attached Security Groups can still be managed using AWS Load Balancer Controller.

Creating a New NLB with AWS Load Balancer Controller

I confirmed the behavior of Security Groups created using annotations. First, prepare an EKS cluster with AWS Load Balancer Controller installed:

eksctl create cluster --name new-nlb

Installation instructions for AWS Load Balancer Controller can be found here.
https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.6/deploy/installation/

Next, prepare the application (nginx) to connect to:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25.2
        ports:
        - containerPort: 80

To create an NLB, create a Service resource. Initially, deploy it with the default settings:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: '2'
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: '10'
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: '2'
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true
    service.beta.kubernetes.io/aws-load-balancer-type: external
  labels:
    app.kubernetes.io/instance: postfix-internal
    app.kubernetes.io/name: nginx
    app.kubernetes.io/version: 1.25.2
  name: nginx
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer
  loadBalancerSourceRanges:
    - [your ip]

This will create an NLB with automatically generated Security Groups.

Additionally, the backend SG rules (TCP:80) allowing communication from the NLB's backend SG to the Node Group's Security Group are automatically added.

Next, I specify an existing Security Group using the service.beta.kubernetes.io/aws-load-balancer-security-groups annotation:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: '2'
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: '10'
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: '2'
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-security-groups: [sg id]
  labels:
    app.kubernetes.io/instance: postfix-internal
    app.kubernetes.io/name: nginx
    app.kubernetes.io/version: 1.25.2
  name: nginx
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

This will attach the specified Security Group (frontend SG) to the NLB. However, in this case, the backend SG is not created automatically. You will need to manually update the Node Group's Security Group to allow communication from the frontend SG.

To automate the creation of the backend SG, attachment to the NLB, and modification of the Node Group's Security Group, set the service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules annotation to 'true'. This annotation defaults to 'false', meaning it won't manage the backend SG when you explicitly specify the frontend SG. Let's deploy the NLB again after deleting it:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: '2'
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: '10'
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: '2'
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-security-groups: [sg id]
    service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: 'true'
  labels:
    app.kubernetes.io/instance: postfix-internal
    app.kubernetes.io/name: nginx
    app.kubernetes.io/version: 1.25.2
  name: nginx
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Now, the specified frontend SG is attached, the backend SG is automatically generated, and the Node Group's Security Group is updated.

Conclusion

The addition of Security Group support for NLB has brought about changes in how NLBs created with AWS Load Balancer Controller behave. While existing setups remain unaffected, for future NLB creations, it's essential to be mindful of the specifications outlined above.