Configuring EKS Managed Node Groups to Use a Proxy with Terraform

In enterprise environments, security and network policies are paramount. It's common practice for servers, including Kubernetes worker nodes, to reside in private subnets with no direct outbound internet access. Instead, all egress traffic must be routed through a centrally managed and monitored HTTP/HTTPS proxy. While this enhances security, it introduces a configuration challenge for services like Amazon EKS, which need to pull container images, communicate with AWS APIs, and perform other bootstrap tasks.

When using the popular terraform-aws-modules/eks/aws module to provision your cluster, you need a reliable, automated way to inject this proxy configuration into your managed node groups. The goal is to ensure that the operating system, the container runtime (containerd), and the Kubernetes components themselves are all proxy-aware from the moment they boot.

This article details a robust solution using the cloudinit_pre_nodeadm hook provided by the module to configure your EKS nodes for a proxy environment.

The core challenge: comprehensive proxy awareness

Simply setting HTTP_PROXY and HTTPS_PROXY environment variables is not enough. A modern EKS node, built on the Amazon Linux 2 EKS Optimized AMI, has several key components that must be configured independently:

1. The Base Operating System: For shell sessions and general system utilities.
2. The Container Runtime (containerd): This is critical. Without proxy settings, containerd will fail to pull any images from public registries like Docker Hub or even private registries outside the VPC, such as Amazon ECR.
3. The EKS Bootstrap Process (nodeadm): The new bootstrapping agent for EKS AMIs needs to communicate with the EKS control plane. This communication should bypass the proxy.
4. Package Managers (yum): If you need to install any additional packages as part of your user data, yum must also be configured to use the proxy.

The solution is to use a cloud-init script that runs before the main EKS bootstrapping logic, ensuring the entire environment is correctly configured before any Kubernetes processes are started.

Understanding the bootstrap hook: cloudinit_pre_nodeadm

Before diving into the code, it's essential to understand the mechanism we're using. The name cloudinit_pre_nodeadm breaks down into two key concepts:

• cloud-init: This is the de facto industry standard for early-stage initialization of cloud instances. When a new EC2 instance boots for the first time, it runs cloud-init, which executes a series of user-provided instructions (user data) to configure the operating system, install packages, create files, and set up users before the machine is considered fully "ready".
• nodeadm: Starting with the EKS Optimized Amazon Linux 2023 AMI, nodeadm is the official bootstrap agent responsible for configuring a node and joining it to an EKS cluster. It handles tasks like retrieving cluster certificates, configuring the kubelet, and applying node labels and taints. It has replaced the legacy bootstrap.sh script.

The cloudinit_pre_nodeadm parameter, provided by the terraform-aws-modules/eks/aws module, is a powerful hook that lets you run a custom cloud-init script at a precise moment: after basic OS initialization but before the nodeadm service is started.

This timing is critical. By executing our script before nodeadm, we ensure that the foundational network environment—including our proxy settings—is already in place. When nodeadm, containerd, and the kubelet eventually start, they inherit the correct configuration from the environment, allowing them to function properly within the restricted network.

The solution: using cloudinit_pre_nodeadm in Terraform

Our solution will pass the Kubernetes service CIDR from our Terraform configuration directly into the user data script. This removes hardcoded values and makes the solution reusable across different clusters.

Here is the Terraform code block that implements the complete proxy configuration.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  # Define the service CIDR for the cluster
  cluster_service_ipv4_cidr = "10.100.0.0/16"

  # ... other EKS cluster configuration ...

  eks_managed_node_groups = {
    main_nodes = {
      # ... other node group configuration like instance_types, min_size, etc. ...

      cloudinit_pre_nodeadm = [
        {
          content_type = "text/x-shellscript"
          content      = <<-EOT
            #!/bin/bash
            set -ex

            # Define your proxy endpoint
            PROXY="http://your-proxy-url:3128" # standard proxy port

            # Use IMDSv2 to securely fetch instance metadata
            TOKEN=`curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
            MAC=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/mac)
            VPC_CIDR=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC/vpc-ipv4-cidr-blocks | xargs | tr ' ' ',')
                        K8S_SERVICE_CIDR="${module.eks.cluster_service_ipv4_cidr}"

            # Define your NO_PROXY list. This is critical for internal and AWS service communication.
            # You might need to add your Pod CIDR as well.
            NO_PROXY="$VPC_CIDR,$K8S_SERVICE_CIDR,localhost,127.0.0.1,169.254.169.254,.amazonaws.com,.svc.cluster.local,.svc,.cluster.local"

            # 1. Configure system-wide proxy settings
            echo "Setting up /etc/environment"
            cat << EOF > /etc/environment
            HTTP_PROXY=$PROXY
            HTTPS_PROXY=$PROXY
            NO_PROXY=$NO_PROXY
            http_proxy=$PROXY
            https_proxy=$PROXY
            no_proxy=$NO_PROXY
            EOF

            # 2. Configure containerd proxy via systemd override
            echo "Configuring containerd service"
            mkdir -p /etc/systemd/system/containerd.service.d
            cat << EOF > /etc/systemd/system/containerd.service.d/http-proxy.conf
            [Service]
            EnvironmentFile=/etc/environment
            EOF

            # 3. Configure nodeadm proxy via systemd override
            echo "Configuring nodeadm service"
            mkdir -p /etc/systemd/system/nodeadm.service.d
            cat << EOF > /etc/systemd/system/nodeadm.service.d/http-proxy.conf
            [Service]
            EnvironmentFile=/etc/environment
            EOF

            # 4. Configure yum proxy
            echo "Configuring yum"
            echo "proxy=$PROXY" >> /etc/yum.conf

            # 5. Reload systemd daemon and restart containerd to apply changes
            echo "Reloading systemd and restarting containerd"
            systemctl daemon-reload
            systemctl restart containerd
          EOT
        }
      ]
    }
  }

  # ... other module configuration ...
}

Dissecting the script

Let's break down the content of the cloud-init script to understand each step.

• set -ex: This is a standard best practice for shell scripting. set -e ensures the script will exit immediately if a command fails, and set -x prints each command before it is executed, providing clear debug output in the system logs.
• Fetching metadata with IMDSv2: The script dynamically fetches the VPC CIDR block directly from the EC2 metadata service. This is crucial for the NO_PROXY variable, ensuring that any traffic destined for other resources within the VPC bypasses the proxy. It uses the more secure IMDSv2 method, which requires a session token.
• Defining NO_PROXY: The NO_PROXY variable is just as important as HTTP_PROXY. It specifies a comma-separated list of domains and IP ranges that should not use the proxy. Our list includes:
  • The dynamically fetched $VPC_CIDR for all internal VPC traffic.
  • The Kubernetes service CIDR passed from module via $K8S_SERVICE_CIDR (e.g., 10.100.0.0/16).
  • localhost and the metadata service address (169.254.169.254).
  • Key AWS and Kubernetes service endpoints to ensure direct communication with the control plane and other AWS APIs. Note the addition of .amazonaws.com to ensure services like ECR, S3, and EC2 are accessed directly via VPC Endpoints if they exist.
• /etc/environment: This file provides the system-wide environment variables for all users and processes. It's the foundation of our configuration.
• Systemd overrides: The modern, correct way to modify a systemd service like containerd or nodeadm is to create an override file. We create http-proxy.conf for both services. The EnvironmentFile=/etc/environment directive instructs systemd to load all variables from our global configuration file before starting the service. This is cleaner and more maintainable than modifying the main service files directly.
• yum.conf: A simple line added to /etc/yum.conf makes the package manager proxy-aware.
• Reload and restart: Finally, systemctl daemon-reload forces systemd to re-read its configuration files, and systemctl restart containerd applies the new environment variables to the container runtime immediately. nodeadm will pick up its configuration when it runs shortly after this script completes.

Successfully deploying EKS worker nodes in a network firewalled behind an HTTP proxy is a common enterprise challenge, but it doesn't have to be a complex one. By leveraging the cloudinit_pre_nodeadm hook within the terraform-aws-modules/eks/aws module, you gain precise control over the node's bootstrap sequence.

This method provides a declarative, automated, and robust solution. By injecting a dynamic script that configures the operating system, containerd, and nodeadm before these critical services start, you ensure nodes join the cluster and pull images reliably. This Infrastructure as Code approach not only solves the immediate technical problem but also creates a repeatable, version-controlled pattern for building secure and compliant Kubernetes platforms on AWS.