DEV Community: César Sepúlveda Barra

I Built a Kubernetes Operator That Programs My Cisco Router

César Sepúlveda Barra — Tue, 24 Feb 2026 20:36:13 +0000

I wrote a Kubernetes operator in Go that talks to a Cisco 4331 router via RESTCONF. It creates VLANs, DHCP pools, and ACLs on the router, all triggered by kubectl apply. Pods get their IPs directly from the router's DHCP server, and inter-VLAN traffic is controlled by real ACLs running on real hardware. This is the full walkthrough.

1. Why?

Kubernetes networking is, by default, flat. Every pod can reach every other pod. That's fine for many workloads, but in plenty of scenarios you actually want segmentation. You want the database on a different network than the web servers. You want firewall rules between them.

There are better tools for this

I want to be honest from the start: if you need network segmentation in Kubernetes today, use Cilium or Calico. They provide NetworkPolicy enforcement, eBPF based segmentation, encryption, observability. They work in software, they scale, and thousands of companies run them in production. That's the right answer for most people.

If you're deep in the Cisco world, ACI with Nexus switches is the official enterprise play. It integrates natively with Kubernetes, gives you policy-driven microsegmentation, multi-tenant networking, full visibility. But it requires Nexus 9000 hardware and APIC controllers, and that's a serious investment.

So why did I do this?

I had a Cisco 4331 router and a Catalyst switch on my desk. Not data center gear. Just mid-range networking equipment, the kind of thing you can pick up on eBay for a couple hundred bucks. The 4331 runs IOS-XE 16.12 and has RESTCONF enabled: a REST API for managing the router configuration over HTTPS.

I wanted to see if I could wire that into Kubernetes. Not through some vendor plugin, but through a custom operator that I wrote from scratch. The idea was simple: define VLANs and policies as Kubernetes CRDs, and let the operator program the router to make them real.

A pod says "I belong to VLAN 10." The operator creates the subinterface on the router, sets up DHCP, writes the ACLs, and the pod gets an IP from the router. No CLI sessions. No separate workflow. Just YAML.

This was never about competing with Cilium or replacing ACI. It was about proving that the operator pattern can extend Kubernetes to control physical network infrastructure, even with equipment that wasn't designed for this.

2. Physical Architecture

This runs on real hardware. No simulations, no GNS3, no virtual routers.

The Lab

Component	Model	IP	Role
Router	Cisco 4331 (IOS-XE 16.12)	192.168.200.1	Gateway, DHCP, ACLs, RESTCONF API
Switch	Cisco Catalyst	192.168.200.2	L2 switching, VLAN trunks
Node 1	Mini PC (8 vCPU, 32 GB)	192.168.200.11	K3s server + worker
Node 2	Mini PC (8 vCPU, 24 GB)	192.168.200.12	K3s server + worker
Node 3	Mini PC (4 vCPU, 16 GB)	192.168.200.13	K3s server + worker

Network Topology

Two NICs per Node

Each node has two network interfaces. This is key to the whole design.

NIC1 (enp1s0): Management. Goes to an access port on the switch, VLAN 1, network 192.168.200.0/24. Carries all the K3s traffic: API server, etcd, Flannel overlay.
NIC2 (enp2s0): Trunk. Goes to a trunk port on the switch that allows VLANs 10, 20, 30. This is the VLAN data plane. Pods attach to this interface via macvlan.

Switch Ports

Ports connected to NIC2 on each node: trunk mode, allowing tagged traffic for VLANs 10, 20, 30
Ports connected to NIC1 on each node: access mode, VLAN 1 (management)
Uplink to router Gi0/0/0: trunk mode (802.1Q), carrying all VLANs

Router on a Stick

The router's GigabitEthernet0/0/0 is a trunk. It has subinterfaces for each VLAN (.10, .20, .30), each one with its own IP, DHCP pool, and ACL. All inter-VLAN traffic goes through the router, where ACLs decide what passes and what gets dropped.

Two Networks per Pod

Every pod ends up with two interfaces:

Pod
├── eth0:  10.42.x.x  (Flannel overlay: K8s API, DNS, Services)
└── net1: 172.16.x.x  (VLAN via macvlan: app data traffic)

Kubernetes internal stuff (API, DNS, service discovery) goes over Flannel. Application traffic between VLANs goes through the physical network, through the router, through real ACLs.

3. Step by Step

Starting from a completely clean cluster. No operator installed, no VLANs, no demo apps. Everything from scratch.

3.1 Install the Operator

helm install cisco-restconf-operator ./charts/cisco-restconf-operator \
    --set image.tag=v0.8.1 \
    --set router.username=admin \
    --set router.password='1234'

NAME: cisco-restconf-operator
LAST DEPLOYED: Tue Feb 24 15:08:32 2026
NAMESPACE: default
STATUS: deployed

One command. That's it. Here's what it created:

Three CRDs registered in the Kubernetes API:

$ kubectl get crd | grep cisco
ciscorouterconfigs.cisco.io    2026-02-24T18:08:32Z
ciscovlans.cisco.io            2026-02-24T18:08:32Z
vlanpolicies.cisco.io          2026-02-24T18:08:32Z

The operator pod plus a DaemonSet running on all three nodes (handles VLAN sub-interfaces and the DHCP CNI daemon):

$ kubectl get pods -n cisco-operator-system
NAME                                                         READY   STATUS
cisco-restconf-operator-controller-manager-74bdd6c5c-x48f2   1/1     Running
cisco-restconf-operator-vlan-setup-6qgdw                     2/2     Running
cisco-restconf-operator-vlan-setup-7dj8f                     2/2     Running
cisco-restconf-operator-vlan-setup-gldg4                     2/2     Running

Two secrets: one with the router credentials, one with TLS certs for the mutating webhook:

$ kubectl get secret -n cisco-operator-system | grep -E "router|webhook"
cisco-restconf-operator-webhook-tls   kubernetes.io/tls   2
cisco-router-credentials              Opaque              2

And here's the interesting part. The Helm chart also creates a CiscoRouterConfig object, and the operator immediately connects to the router via RESTCONF:

$ kubectl get ciscorouterconfigs -o wide
NAME      HOST            CONNECTED   MESSAGE                                  AGE
default   192.168.200.1   true        Connected to LAB-ROUTER (IOS-XE 16.12)   26s

Look at the status:

status:
  connected: true
  hostname: LAB-ROUTER
  lastConnected: "2026-02-24T18:08:35Z"
  message: Connected to LAB-ROUTER (IOS-XE 16.12)
  version: "16.12"

The operator is alive, talking to the router, and reporting back its hostname and firmware version. It will re-validate the connection every 5 minutes.

3.2 Create VLANs

Three VLANs for the demo:

# 00-vlans.yaml
apiVersion: cisco.io/v1alpha1
kind: CiscoVLAN
metadata:
  name: vlan-10
spec:
  vlanId: 10
  cidr: "172.16.10.0/24"
  gateway: "172.16.10.1"
  dhcpRange:
    start: "172.16.10.100"
    end: "172.16.10.200"
---
apiVersion: cisco.io/v1alpha1
kind: CiscoVLAN
metadata:
  name: vlan-20
spec:
  vlanId: 20
  cidr: "172.16.20.0/24"
  gateway: "172.16.20.1"
  dhcpRange:
    start: "172.16.20.100"
    end: "172.16.20.200"
---
apiVersion: cisco.io/v1alpha1
kind: CiscoVLAN
metadata:
  name: vlan-30
spec:
  vlanId: 30
  cidr: "172.16.30.0/24"
  gateway: "172.16.30.1"
  dhcpRange:
    start: "172.16.30.100"
    end: "172.16.30.200"

$ kubectl apply -f 00-vlans.yaml
ciscovlan.cisco.io/vlan-10 created
ciscovlan.cisco.io/vlan-20 created
ciscovlan.cisco.io/vlan-30 created

A few seconds later, all three are Active:

$ kubectl get ciscovlans -o wide
NAME      VLAN-ID   CIDR             GATEWAY       STATE    PODS   AGE
vlan-10   10        172.16.10.0/24   172.16.10.1   Active          18s
vlan-20   20        172.16.20.0/24   172.16.20.1   Active          18s
vlan-30   30        172.16.30.0/24   172.16.30.1   Active          18s

The status on each VLAN tells you exactly what got created on the router:

status:
  state: Active
  routerSubinterface: GigabitEthernet0/0/0.10
  message: VLAN 10 configured on GigabitEthernet0/0/0.10
  lastReconciled: "2026-02-24T18:09:13Z"

So what actually happened on the router? I queried it via RESTCONF to confirm.

Subinterfaces created, each one with its own IP and ip nat inside:

GigabitEthernet0/0/0       (trunk, no IP)
GigabitEthernet0/0/0.10    172.16.10.1/24
GigabitEthernet0/0/0.20    172.16.20.1/24
GigabitEthernet0/0/0.30    172.16.30.1/24

DHCP pools, one per VLAN:

{ "id": "VLAN10_POOL", "network": "172.16.10.0/24", "default-router": "172.16.10.1" }
{ "id": "VLAN20_POOL", "network": "172.16.20.0/24", "default-router": "172.16.20.1" }
{ "id": "VLAN30_POOL", "network": "172.16.30.0/24", "default-router": "172.16.30.1" }

Base ACLs. This is important. By default, each VLAN can talk to itself and to the internet, but is blocked from reaching any other VLAN:

VLAN10_ACL:
  10  permit ip   172.16.10.0 0.0.0.255 -> 172.16.10.0 0.0.0.255    (intra-VLAN ok)
  20  deny   ip   172.16.10.0 0.0.0.255 -> 172.16.20.0 0.0.0.255    (block VLAN 20)
  30  deny   ip   172.16.10.0 0.0.0.255 -> 172.16.30.0 0.0.0.255    (block VLAN 30)
1000  permit ip   any -> any                                          (internet ok)

VLAN20_ACL:
  10  permit ip   172.16.20.0 0.0.0.255 -> 172.16.20.0 0.0.0.255
  20  deny   ip   172.16.20.0 0.0.0.255 -> 172.16.10.0 0.0.0.255
  30  deny   ip   172.16.20.0 0.0.0.255 -> 172.16.30.0 0.0.0.255
1000  permit ip   any -> any

VLAN30_ACL:
  10  permit ip   172.16.30.0 0.0.0.255 -> 172.16.30.0 0.0.0.255
  20  deny   ip   172.16.30.0 0.0.0.255 -> 172.16.10.0 0.0.0.255
  30  deny   ip   172.16.30.0 0.0.0.255 -> 172.16.20.0 0.0.0.255
1000  permit ip   any -> any

One kubectl apply. The router now has subinterfaces, DHCP pools, and ACLs for three VLANs. I didn't type a single command on the router CLI.

3.3 Deploy Applications (Before Policy)

The demo uses three pods:

postgres on VLAN 20: a PostgreSQL database
hello-world on VLAN 10: a Flask app that tries to connect to PostgreSQL
no-hello-world on VLAN 30: the exact same Flask app, but on a different VLAN

# 04-apps.yaml (simplified)
apiVersion: v1
kind: Pod
metadata:
  name: hello-world
  namespace: demo
  annotations:
    cisco.io/vlan: vlan-10          # <-- This is all you need
spec:
  containers:
    - name: hello-world
      image: 192.168.100.254:5000/hello-world:v1
      env:
        - name: DB_HOST
          value: "postgres-vlan.demo.svc.cluster.local"

The pod only has cisco.io/vlan: vlan-10. A mutating webhook intercepts pod creation and injects the Multus network annotation automatically. The app uses DNS (postgres-vlan.demo.svc.cluster.local) to find the database. The operator creates a headless Service that resolves to the database's DHCP-assigned VLAN IP.

$ kubectl apply -f 04-apps.yaml
namespace/demo created
pod/postgres created
pod/hello-world created
pod/no-hello-world created

$ kubectl get pods -n demo -o wide
NAME             READY   STATUS    IP           NODE
hello-world      1/1     Running   10.42.1.97   node-2
no-hello-world   1/1     Running   10.42.1.98   node-2
postgres         1/1     Running   10.42.0.93   node-1

The VLAN CRDs now show one active pod each:

$ kubectl get ciscovlans -o wide
NAME      VLAN-ID   CIDR             GATEWAY       STATE    PODS
vlan-10   10        172.16.10.0/24   172.16.10.1   Active   1
vlan-20   20        172.16.20.0/24   172.16.20.1   Active   1
vlan-30   30        172.16.30.0/24   172.16.30.1   Active   1

I port-forwarded both Flask apps to my laptop:

kubectl port-forward pod/hello-world -n demo 8080:8080 &
kubectl port-forward pod/no-hello-world -n demo 8081:8080 &

Open http://localhost:8080 and http://localhost:8081. Both apps have a dashboard that auto-refreshes every second, showing whether the database is reachable.

Both show DB UNREACHABLE. Red badge on both.

// hello-world (VLAN 10) at http://localhost:8080/api/status
{
    "db_reachable": false,
    "db_error": "connection failed: No route to host",
    "net1_ip": "172.16.10.2",
    "pod": "hello-world"
}

// no-hello-world (VLAN 30) at http://localhost:8081/api/status
{
    "db_reachable": false,
    "db_error": "connection failed: No route to host",
    "net1_ip": "172.16.30.2",
    "pod": "no-hello-world"
}

That's the base ACL doing its job. VLANs 10 and 30 both have deny rules blocking traffic to VLAN 20 where PostgreSQL lives. The segmentation is real, enforced on the router hardware, not in software.

3.4 Apply a VLANPolicy

Now let's selectively open access. We want hello-world (VLAN 10) to reach PostgreSQL (VLAN 20) on TCP 5432, but keep no-hello-world (VLAN 30) blocked.

# 01-policy.yaml
apiVersion: cisco.io/v1alpha1
kind: VLANPolicy
metadata:
  name: app-to-db
spec:
  source: vlan-10
  destination: vlan-20
  rules:
    - protocol: tcp
      ports: [5432]

$ kubectl apply -f 01-policy.yaml
vlanpolicy.cisco.io/app-to-db created

$ kubectl get vlanpolicies -o wide
NAME        SOURCE    DESTINATION   STATE     ACL-ENTRIES   AGE
app-to-db   vlan-10   vlan-20       Applied   10            9s

The operator updated both ACLs on the router:

status:
  state: Applied
  aclEntries: 10
  message: "ACL rules applied: VLAN10_ACL (5 entries), VLAN20_ACL (5 entries)"

Here's what the ACLs look like now on the router. The operator inserted permit rules before the existing deny rules:

VLAN10_ACL (after policy):
  10  permit tcp  172.16.10.0 -> 172.16.20.0 eq 5432    << NEW: allow TCP 5432
  20  permit ip   172.16.10.0 -> 172.16.10.0             (intra-VLAN)
  30  deny   ip   172.16.10.0 -> 172.16.20.0             (block everything else to VLAN 20)
  40  deny   ip   172.16.10.0 -> 172.16.30.0             (block VLAN 30)
1000  permit ip   any -> any                              (internet)

VLAN20_ACL (after policy):
  10  permit tcp  172.16.20.0 -> 172.16.10.0 established << NEW: return traffic only
  20  permit ip   172.16.20.0 -> 172.16.20.0
  30  deny   ip   172.16.20.0 -> 172.16.10.0
  40  deny   ip   172.16.20.0 -> 172.16.30.0
1000  permit ip   any -> any

Two things to notice:

VLAN10_ACL now has a permit tcp ... eq 5432 at sequence 10, before the deny. VLAN 10 can initiate TCP connections to VLAN 20 on the PostgreSQL port.
VLAN20_ACL has an established rule. This allows return traffic from connections that VLAN 10 started, but VLAN 20 cannot initiate new connections to VLAN 10.

VLAN30_ACL didn't change at all. No policy mentions VLAN 30, so it stays fully blocked.

3.5 The Result

Back to the browsers. The dashboards refresh every second.

hello-world (VLAN 10) at http://localhost:8080

{
    "db_reachable": true,
    "db_host": "postgres-vlan.demo.svc.cluster.local",
    "latency_ms": 7.2,
    "net1_ip": "172.16.10.2",
    "pod": "hello-world"
}

Green badge. DB REACHABLE, 7.2ms. It can also write and read data:

// GET http://localhost:8080/db
{
    "status": "SUCCESS",
    "inserted": {
        "id": 1,
        "created_at": "2026-02-24T18:14:04.158103"
    },
    "recent_messages": [
        {
            "content": "Hello from 172.16.10.2 at 18:14:04",
            "id": 1
        }
    ]
}

no-hello-world (VLAN 30) at http://localhost:8081

{
    "db_reachable": false,
    "db_error": "connection failed: timeout expired",
    "net1_ip": "172.16.30.2",
    "pod": "no-hello-world"
}

Red badge. DB UNREACHABLE. Same Docker image. Same code. Same DB_HOST env var. The only difference is one line in the YAML: cisco.io/vlan: vlan-30 instead of vlan-10. The router drops the traffic.

// GET http://localhost:8081/db
{
    "status": "ERROR",
    "error": "connection failed: No route to host"
}

4. What's Next

The full source code is on GitHub: github.com/csepulveda/cisco-kubernetes

Let me repeat what I said at the beginning: this is not production software. The code is tied to a specific router model (Cisco 4331), a specific IOS-XE version (16.12), and a specific physical setup. If you actually need network segmentation in Kubernetes, go with Cilium or Calico. If you're in the Cisco ecosystem and have the budget, ACI with Nexus is what it's designed for. Those are real, tested, supported solutions.

This project is something else. It's a lab experiment. A proof of concept. And the thing it proves is this: Kubernetes operators can manage anything that has an API.

The operator pattern is a control loop. It watches for desired state (your CRDs) and makes actual state match. Most operators manage stuff inside the cluster: databases, message queues, certificates. But nothing says it has to stay inside. The reconciliation loop doesn't care whether it's creating a Pod or configuring a physical router 3 meters away.

In this case it's a Cisco router. But it could be a firewall, a load balancer, a DNS provider, a cloud networking service, some legacy system with SOAP endpoints. If it has an API and it has state, you can write an operator for it.

That's what I find exciting about this. I wired a mid-range Cisco 4331 (not a Nexus 9000, not an ACI fabric, just a regular router) into a Kubernetes control plane using nothing but RESTCONF and Go. No vendor SDK, no proprietary integration, no expensive hardware. Just the operator pattern doing what it does best: turning YAML into reality.

Built with Go, Kubebuilder, K3s, and a lot of patience debugging YANG models at 2am.

From Zero to EKS and Hybrid-Nodes —Part 3: Setting up the NLB, Ingress, and Services on a Hybrid EKS Infrastructure

César Sepúlveda Barra — Mon, 21 Apr 2025 03:41:08 +0000

Introduction:

In the previous part, we set up the EKS cluster and deployed the first hybrid nodes. At that point, the network topology looked like this:

Now, in this third part, we’re going to set up the Network Load Balancer (NLB), configure Ingress, and deploy services to expose our application running on this hybrid infrastructure. We’ll also observe and analyze the behavior of a Kubernetes Deployment in this mixed node environment.

Network Topology Changes:

To enable communication between pods running on AWS-managed nodes and those on hybrid nodes, we need to configure specific routing rules on our VPN gateway/router.

In a production environment, you could enable BGP in Cilium (if your network supports it), but since this is a lab setup, we’ll handle routing manually via static routes on the gateway.

Disabling kube-proxy on Hybrid Nodes:

Since we’re replacing kube-proxy functionality with Cilium, we’ll prevent kube-proxy from running on hybrid nodes using the following patch:

kubectl patch daemonset kube-proxy \
  -n kube-system \
  --type='json' \
  -p='[
    {
      "op": "add",
      "path": "/spec/template/spec/affinity/nodeAffinity/requiredDuringSchedulingIgnoredDuringExecution/nodeSelectorTerms/0/matchExpressions/2/values/-",
      "value": "hybrid"
    }
  ]'

Get EKS API Server Endpoint:

This will be needed in the next step:

aws eks describe-cluster --name my-eks-cluster --query "cluster.endpoint"

Updating the Cilium Configuration:

Next, we’ll update our cilium-values.yaml file to:

Enable kubeProxyReplacement.
Enable Layer 2 (L2) announcements.
Define the /25 subnets to be used for pod IPs.
Set the Kubernetes API server endpoint and port.

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: eks.amazonaws.com/compute-type
          operator: In
          values:
          - hybrid
ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4MaskSize: 25
    clusterPoolIPv4PodCIDRList:
    - 192.168.101.0/25
    - 192.168.101.128/25
operator:
  replicas: 1
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: eks.amazonaws.com/compute-type
            operator: In
            values:
              - hybrid
  unmanagedPodWatcher:
    restart: false
envoy:
  enabled: false
kubeProxyReplacement: true
l2announcements:
  leaseDuration: 2s
  leaseRenewDeadline: 1s
  leaseRetryPeriod: 200ms
  enabled: true
externalIPs:
  enabled: true

k8sClientRateLimit:
  qps: 100
  burst: 150

k8sServiceHost: <CLUSTER_HOST>
k8sServicePort: 443

Apply the changes:

cd commands/02-cilium-updates
helm upgrade -i cilium cilium/cilium -n kube-system -f cilium-values.yaml

Verifying Cilium Agents:

You should now see cilium-agent pods running on the hybrid nodes, and no kube-proxy pods there:

kubectl -n kube-system get pods -o wide
NAME                               READY   STATUS    RESTARTS   AGE   IP                NODE                         NOMINATED NODE   READINESS GATES
aws-node-k8qw5                     2/2     Running   0          31m   10.0.21.61        ip-10-0-21-61.ec2.internal   <none>           <none>
cilium-lxphl                       1/1     Running   0          48s   192.168.100.102   mi-00a55e151790a4c81         <none>           <none>
cilium-operator-7658767979-f57v5   1/1     Running   0          52s   192.168.100.101   mi-014887c08038cb6c2         <none>           <none>
cilium-znwdm                       1/1     Running   0          49s   192.168.100.101   mi-014887c08038cb6c2         <none>           <none>
coredns-6b9575c64c-fxzv9           1/1     Running   0          35m   10.0.25.252       ip-10-0-21-61.ec2.internal   <none>           <none>
eks-pod-identity-agent-k559t       1/1     Running   0          31m   10.0.21.61        ip-10-0-21-61.ec2.internal   <none>           <none>
kube-proxy-6j74v                   1/1     Running   0          15m   10.0.21.61        ip-10-0-21-61.ec2.internal   <none>           <none>

Static Routing:

Now it’s time to configure static routes on the VPN router. First, check which hybrid node is responsible for each /25 subnet:

kubectl get ciliumnodes
NAME                   CILIUMINTERNALIP   INTERNALIP        AGE
mi-00a55e151790a4c81   192.168.101.51     192.168.100.102   8m14s
mi-014887c08038cb6c2   192.168.101.192    192.168.100.101   8m14s

Set the routes:

ip route add 192.168.101.0/25 via 192.168.100.102
ip route add 192.168.101.128/25 via 192.168.100.101

Now out network will look like this:

Installing the Load Balancer Controller:

Add the following Terraform file to your project to install the AWS Load Balancer Controller via Helm, and create the required IAM role:

resource "helm_release" "aws-load-balancer-controller" {
  name       = "aws-load-balancer-controller"
  chart      = "aws-load-balancer-controller"
  repository = "https://aws.github.io/eks-charts"
  namespace  = "kube-system"
  version    = "1.12.0"

  set {
    name  = "serviceAccount.create"
    value = "true"
  }
  set {
    name  = "serviceAccount.name"
    value = module.eks_loadbalancer_iam.iam_role_name
  }
  set {
    name  = "serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
    value = module.eks_loadbalancer_iam.iam_role_arn
  }
  set {
    name  = "clusterName"
    value = module.eks.cluster_name
  }
  set {
    name  = "replicaCount"
    value = "1"
  }
  set {
    name  = "vpcId"
    value = data.terraform_remote_state.vpc.outputs.vpc_id
  }

  # Optional affinity rule
  set {
    name  = "affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key"
    value = "eks.amazonaws.com/capacityType"
  }
  set {
    name  = "affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].operator"
    value = "Exists"
  }
}

module "eks_loadbalancer_iam" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "5.54.0"

  role_name                              = "load-balancer-controller"
  attach_load_balancer_controller_policy = true

  oidc_providers = {
    ex = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["kube-system:load-balancer-controller"]
    }
  }
}

Deploy it:

cd ../../EKS-HYBRID
cp ../load-balancer/load-balancer.tf .
tofu init
tofu apply

Verify the changes:

kubectl get deployment/aws-load-balancer-controller -n kube-system
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
aws-load-balancer-controller   1/1     1            1           5m

Deploying the Demo App:

Now let’s deploy a simple app with 6 replicas to observe pod distribution across AWS and hybrid nodes.

apiVersion: v1
kind: Namespace
metadata:
  name: whoami
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  namespace: whoami
spec:
  replicas: 6
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: whoami
      containers:
        - image: containous/whoami
          imagePullPolicy: Always
          name: whoami
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: whoami
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 80
      targetPort: 80
  selector:
    app: whoami
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami
  namespace: whoami
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}]'
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/load-balancer-type: nlb
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: whoami
                port:
                  number: 80

Apply it:

cd ../commands/03-create-service
kubectl apply -f service.yaml

Check pod distribution:

kubectl get pods -n whoami -o wide
NAME                     READY   STATUS    RESTARTS   AGE    IP                NODE                         NOMINATED NODE   READINESS GATES
whoami-7cb8f48c8-bhrvw   1/1     Running   0          2m6s   192.168.101.187   mi-014887c08038cb6c2         <none>           <none>
whoami-7cb8f48c8-gd48g   1/1     Running   0          6s     10.0.21.186       ip-10-0-21-61.ec2.internal   <none>           <none>
whoami-7cb8f48c8-hhs49   1/1     Running   0          2m9s   192.168.101.212   mi-014887c08038cb6c2         <none>           <none>
whoami-7cb8f48c8-n64gh   1/1     Running   0          2m9s   192.168.101.108   mi-00a55e151790a4c81         <none>           <none>
whoami-7cb8f48c8-q8bhj   1/1     Running   0          2m9s   192.168.101.56    mi-00a55e151790a4c81         <none>           <none>
whoami-7cb8f48c8-rr7kp   1/1     Running   0          9s     10.0.27.231       ip-10-0-21-61.ec2.internal   <none>           <none>

Test Round-Robin Behavior:

Get the Ingress address:

kubectl get ingress  -n whoami 
NAME     CLASS   HOSTS   ADDRESS                                                               PORTS   AGE
whoami   alb     *       k8s-whoami-whoami-xxxx-yyyy.us-east-1.elb.amazonaws.com   80      36s

Then run:

for i in $(seq 1 10); 
  do curl -s http://<your-lb-dns>/api | jq .ip 
; done
[
  "127.0.0.1",
  "::1",
  "192.168.101.212",
  "fe80::b405:92ff:fe92:72f4"
]
[
  "127.0.0.1",
  "::1",
  "192.168.101.212",
  "fe80::b405:92ff:fe92:72f4"
]
[
  "127.0.0.1",
  "::1",
  "10.0.21.186",
  "fe80::6c8f:6aff:fe38:5e55"
]
[
  "127.0.0.1",
  "::1",
  "10.0.27.231",
  "fe80::a8a6:26ff:feed:d649"
]
[
  "127.0.0.1",
  "::1",
  "192.168.101.108",
  "fe80::28bc:50ff:fe48:d3d9"
]
[
  "127.0.0.1",
  "::1",
  "192.168.101.56",
  "fe80::7869:ccff:fee5:b61b"
]
[
  "127.0.0.1",
  "::1",
  "192.168.101.187",
  "fe80::14f6:88ff:fe17:f9ae"
]
[
  "127.0.0.1",
  "::1",
  "192.168.101.212",
  "fe80::b405:92ff:fe92:72f4"
]
[
  "127.0.0.1",
  "::1",
  "10.0.21.186",
  "fe80::6c8f:6aff:fe38:5e55"
]
[
  "127.0.0.1",
  "::1",
  "10.0.27.231",
  "fe80::a8a6:26ff:feed:d649"
]

You should see requests being served by both AWS and hybrid nodes.

Pricing Overview:

Hybrid nodes do not incur EC2 costs, but you’ll still pay:

EKS control plane fees
Data transfer
Hybrid node usage based on vCPU-hours

Example:

A 32 vCPU hybrid node = 32 × 24 × 30 = 23,040 vCPU-hours/month
At $0.02 per vCPU-hour, that’s ~$460.80/month
The cheapest AWS instance with 32 vCPU and GPU (e.g. g5g.xlarge) costs ~$987/month — nearly 2x more expensive

Conclusion:

As you can see, it’s entirely possible to run a hybrid EKS environment combining AWS-managed and on-prem nodes. While this is not a common setup, it’s a powerful tool for organizations with existing infrastructure or specific compliance requirements.

This lab setup uses static routing, eks_managed_node_groups, and minimal redundancy. In a production scenario, you’d likely adopt:

VPC CNI custom networking
Karpenter for dynamic scaling
BGP routing for flexibility
A real VPN appliance

But as a proof of concept, it offers a strong foundation and a practical look into what’s required to implement EKS hybrid clusters successfully.

All the files used by this lab, could be found here

Tools I Use Daily as a DevOps Engineer

César Sepúlveda Barra — Fri, 11 Apr 2025 17:14:40 +0000

This is a post I plan to update frequently. Here’s the initial list of top tools I use in my day-to-day work — whether I'm managing Kubernetes clusters, working with AWS, or just trying to stay efficient.

🔧 1. K9s

K9s is a must-have for Kubernetes administrators. It's lightweight, fast, and lets you:

Check and explore Kubernetes resources
Create port-forwardings
Modify, delete, or view logs from pods
And much more

Install: brew install k9s

Demo: YouTube – K9s in Action

🧭 2. kubectx & kubens

kubectx and kubens are simple tools to manage multiple Kubernetes contexts and namespaces efficiently.

Install: brew install kubectx

Docs: GitHub – kubectx

💣 3. aws-nuke

aws-nuke helps you clean up AWS environments easily after testing or PoCs. Just define which IAM resources to keep — and nuke everything else.

Install: brew install aws-nuke

Docs: GitHub – aws-nuke

💸 4. eks-node-viewer

A tool I use almost daily to check how many EKS nodes I'm running and estimate my spending.

Install:

brew tap aws/tap && brew install eks-node-viewer
Docs: GitHub – eks-node-viewer

If you found this list helpful, let me know! I’ll keep this post updated with new tools as I incorporate them into my workflow.

Stay tuned — more tools coming soon!

From Zero to EKS and Hybrid-Nodes — Part 2: The EKS and Hybrid Nodes configuration.

César Sepúlveda Barra — Fri, 11 Apr 2025 13:33:25 +0000

Introduction:

In the previous part, we set up the VPC and the VPN Site-to-Site connection to prepare for our EKS cluster and hybrid nodes. The network diagram at that stage looked like this:

In this second part, we’ll configure the EKS cluster using OpenTofu, including SSM Activation, and manually set up two Hybrid nodes using nodeadm.

Setup the EKS cluster

Cluster creation is straightforward. We’ll use the state file from the VPC creation to retrieve outputs, and we’ll create the cluster using the terraform-aws-modules/eks/aws module.

Full code is available here: https://github.com/csepulveda/modular-aws-resources/tree/main/EKS-HYBRID

################################################################################
# EKS Module
################################################################################
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "20.35.0"

  cluster_name    = local.name
  cluster_version = var.eks_version

  enable_cluster_creator_admin_permissions = true
  cluster_endpoint_public_access           = true

  cluster_addons = {
    coredns = {
      configuration_values = jsonencode({
        replicaCount = 1
      })
    }
    eks-pod-identity-agent = {}
    kube-proxy             = {}
    vpc-cni                = {}
  }

  vpc_id                   = data.terraform_remote_state.vpc.outputs.vpc_id
  subnet_ids               = data.terraform_remote_state.vpc.outputs.private_subnets
  control_plane_subnet_ids = data.terraform_remote_state.vpc.outputs.control_plane_subnet_ids

  eks_managed_node_groups = {
    eks-base = {
      ami_type       = "AL2023_x86_64_STANDARD"
      instance_types = ["t3.small", "t3a.small"]

      min_size      = 1
      max_size      = 1
      desired_size  = 1
      capacity_type = "SPOT"
      network_interfaces = [{
        delete_on_termination = true
      }]
    }
  }

  node_security_group_additional_rules = {
    allow-all-80-traffic-from-loadbalancers = {
      cidr_blocks = [for s in data.aws_subnet.elb_subnets : s.cidr_block]
      description = "Allow all traffic from load balancers"
      from_port   = 80
      to_port     = 80
      protocol    = "TCP"
      type        = "ingress"
    }
    hybrid-all = {
      cidr_blocks = ["192.168.100.0/23"]
      description = "Allow all traffic from remote node/pod network"
      from_port   = 0
      to_port     = 0
      protocol    = "all"
      type        = "ingress"
    }
  }

  cluster_security_group_additional_rules = {
    hybrid-all = {
      cidr_blocks = ["192.168.100.0/23"]
      description = "Allow all traffic from remote node/pod network"
      from_port   = 0
      to_port     = 0
      protocol    = "all"
      type        = "ingress"
    }
  }

  cluster_remote_network_config = {
    remote_node_networks = {
      cidrs = ["192.168.100.0/24"]
    }
    remote_pod_networks = {
      cidrs = ["192.168.101.0/24"]
    }
  }

  access_entries = {
    hybrid-node-role = {
      principal_arn = module.eks_hybrid_node_role.arn
      type          = "HYBRID_LINUX"
    }
  }


  node_security_group_tags = merge(local.tags, {
    "karpenter.sh/discovery" = local.name
  })

  tags = local.tags
}

################################################################################
# Hybrid nodes Support
################################################################################
module "eks_hybrid_node_role" {
  source = "terraform-aws-modules/eks/aws//modules/hybrid-node-role"
  version = "20.35.0"

  name = "hybrid"

  tags = local.tags
}


resource "aws_ssm_activation" "this" {
  name               = "hybrid-node"
  iam_role           = module.eks_hybrid_node_role.name
  registration_limit = 10

  tags = local.tags
}

resource "local_file" "nodeConfig" {
  content  = <<-EOT
    apiVersion: node.eks.aws/v1alpha1
    kind: NodeConfig
    spec:
      cluster:
        name: ${module.eks.cluster_name}
        region: ${local.region}
      hybrid:
        ssm:
          activationId: ${aws_ssm_activation.this.id}
          activationCode: ${aws_ssm_activation.this.activation_code} 
  EOT
  filename = "nodeConfig.yaml"
}

Highlights of the configuration:

One EKS-managed node group for core services and ACK controllers (to be installed in Part 3).
VPC CNI enabled via cluster_addons.
Ingress rules to allow traffic from on-premise networks.
Defined remote node and pod networks.
Created a role for hybrid nodes and granted access via access_entries.

Hybrid Nodes and SSM Activation

We define:

An IAM role using the hybrid node module.
An SSM Activation to allow hybrid nodes to join the cluster.
A nodeConfig.yaml file containing the activation credentials, automatically generated during tofu apply.

tofu apply
....
Apply complete! Resources: 50 added, 0 changed, 0 destroyed.

This generates a nodeConfig.yaml file, which is critical for authenticating on-prem nodes via AWS Systems Manager.

Update your kubeconfig after creation:

aws eks update-kubeconfig --region us-east-1 --name my-eks-cluster

Now your infrastructure should look like this:

Setting Up the Hybrid Nodes

SSH into the nodes (e.g., 192.168.100.101 and 192.168.100.102). These are configured with two networks:

192.168.100.0/24 for nodes.
192.168.101.0/24 for pods.

Transfer the node config:

scp nodeConfig.yaml cesar@192.168.100.101:/tmp/
scp nodeConfig.yaml cesar@192.168.100.102:/tmp/

Install and configure nodeadm:

curl -OL 'https://hybrid-assets.eks.amazonaws.com/releases/latest/bin/linux/arm64/nodeadm'
chmod a+x nodeadm
mv nodeadm /usr/local/bin/.

nodeadm install 1.32 --credential-provider ssm
nodeadm init --config-source file:///tmp/nodeConfig.yaml

Repeat for both nodes.

Once done, check node registration:

kubectl get nodes -o wide
NAME                          STATUS     ROLES    AGE     VERSION               INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION                    CONTAINER-RUNTIME
ip-10-0-19-137.ec2.internal   Ready      <none>   16m     v1.32.1-eks-5d632ec   10.0.19.137       <none>        Amazon Linux 2023.7.20250331   6.1.131-143.221.amzn2023.x86_64   containerd://1.7.27
mi-03c1eb4c6173151d6          NotReady   <none>   3m4s    v1.32.1-eks-5d632ec   192.168.100.102   <none>        Ubuntu 22.04.5 LTS             5.15.0-136-generic                containerd://1.7.24
mi-091f1dddb980a80ff          NotReady   <none>   3m51s   v1.32.1-eks-5d632ec   192.168.100.101   <none>        Ubuntu 22.04.5 LTS             5.15.0-136-generic                containerd://1.7.24

The hybrid nodes appear but are NotReady — this is expected until we install a network controller.

Disclaimer time: Although the documentation claims compatibility with Ubuntu 22.04 and 24.04, I couldn’t get 24.04 working due to missing nf_conntrack when kube-proxy attempts to apply iptables rules. If you manage to fix this, please share!

Setting Up the Network with Cilium

We’ll use Cilium as CNI.

cilium-values.yaml:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: eks.amazonaws.com/compute-type
          operator: In
          values:
          - hybrid
ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4MaskSize: 24
    clusterPoolIPv4PodCIDRList:
    - 192.168.101.0/24
operator:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: eks.amazonaws.com/compute-type
            operator: In
            values:
              - hybrid
  unmanagedPodWatcher:
    restart: false
envoy:
  enabled: false

This yaml file indicate the affinity on cilium pods and also the cilium operator to run only en hybrid nodes, also we set the clusterPoolIPv4PodCIDRList pool, using the subnet 192.168.101.0/24

Install Cilium:

helm repo add cilium https://helm.cilium.io/

helm upgrade -i cilium cilium/cilium \
--version 1.17.2 \
--namespace kube-system \
--values cilium-values.yaml

...
Release "cilium" does not exist. Installing it now.
NAME: cilium
LAST DEPLOYED: Fri Apr 11 08:35:31 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble.

Your release version is 1.17.2.

For any further help, visit https://docs.cilium.io/en/v1.17/gettinghelp

After a few minutes the nodes will be ready

kubectl get nodes -o wide

You should see:

NAME                          STATUS   ROLES    AGE   VERSION               INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION                    CONTAINER-RUNTIME
ip-10-0-19-137.ec2.internal   Ready    <none>   56m   v1.32.1-eks-5d632ec   10.0.19.137       <none>        Amazon Linux 2023.7.20250331   6.1.131-143.221.amzn2023.x86_64   containerd://1.7.27
mi-03c1eb4c6173151d6          Ready    <none>   42m   v1.32.1-eks-5d632ec   192.168.100.102   <none>        Ubuntu 22.04.5 LTS             5.15.0-136-generic                containerd://1.7.24
mi-091f1dddb980a80ff          Ready    <none>   43m   v1.32.1-eks-5d632ec   192.168.100.101   <none>        Ubuntu 22.04.5 LTS             5.15.0-136-generic                containerd://1.7.24

Your hybrid nodes are now fully integrated with EKS. 🎊

Now your infrastructure should look like this:

What’s Next?

In the final part of this series, we’ll:

Deploy the Network Load Balancer (NLB) Controller.
Set up an Ingress.
Deploy a sample service running entirely on hybrid/on-prem nodes.

From Zero to EKS and Hybrid-Nodes — Part 1: The VPC and VPN configuration.

César Sepúlveda Barra — Wed, 09 Apr 2025 14:07:49 +0000

In this post, I’ll walk you through how to create an Amazon EKS node running Kubernetes 1.32 and set up a Site-to-Site VPN to connect an on-prem virtual machine to your EKS cluster using Linux hybrid nodes. To complete the setup, we’ll deploy a Network Load Balancer (NLB) and an NGINX Ingress controller to manage traffic across both cloud-based and on-prem applications.

The goal is to demonstrate how we can extend our on-prem infrastructure into EKS, enabling a seamless hybrid environment. This approach is especially useful in scenarios where you want to leverage local GPU machines for LLM workloads, or manage CDN nodes running on data center hardware while orchestrating everything through Kubernetes.

Create VPC

We’ll start by creating the VPC, which is quite straightforward using OpenTofu (formerly Terraform). Below is the configuration using the terraform-aws-modules/vpc/aws module:

The code is here: https://github.com/csepulveda/modular-aws-resources/tree/main/VPC

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.19.0"

  name = local.name #my-eks-cluster
  cidr = local.vpc_cidr #10.0.0.0/16

  azs             = local.azs #["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)]
  public_subnets  = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)]
  intra_subnets   = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 52)]

  enable_nat_gateway = true
  single_nat_gateway = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
    "karpenter.sh/discovery" = local.name
  }

  tags = local.tags
}

To apply the changes, run:

tofu apply

Once applied, you should see an output similar to the following:

This setup creates a basic but functional VPC with three subnet types, each serving a specific purpose:

Private Subnets: Used for deploying nodes and pods.
Public Subnets: Used for exposing services via Network Load Balancers.
Intra Subnets: Dedicated to the EKS control plane. In the next steps, we’ll manually configure routing and VPN components. I’m choosing not to automate those with Terraform to better explain each concept and step in detail.

VPN Site to Site:

As you could see in the infra diagram, the local network CIDR is 192.168.100.0/23 and the VPC CIDR is 10.0.0.0/16 so we need to go to each subnet in our new VPC

Create the Customer gateway.

First we need to create the Customer gateway, this is a very simple step, we only need to define a name (Name tag — optional) and also what is the public IP (IP address) address of our local network router (in my case the routes is in a DMZ behind a NAT)

Name tag: eks-kybrid-customer-gateway
IP Address: [Your public IPv4]

Create the Virtual private gateway

To create the Virtual Private Gateway, you only need to assign a name tag. This component will act as the gateway between your AWS VPC and the on-premises network.

Name tag: eks-kybrid-private-gateway

Create the Site-To-Site VPN connection

Once the Customer Gateway and Virtual Private Gateway are created, we can proceed to establish the Site-to-Site VPN connection.

In this step, we will configure the following settings:
Name tag: eks-kybrid-vpn
Target gateway type: Virtual Private Gateway (select the one created earlier)
Customer gateway: Existing (select the previously created Customer Gateway)
Routing options: Static
Static IP prefixes: 192.168.100.0/23 (your on-prem network)
Local IPv4 network CIDR: 192.168.100.0/23 (your on-prem network)
Remote IPv4 network CIDR: 10.0.0.0/16 (the VPC CIDR)

This configuration defines the routing paths for traffic between your on-premises network and the EKS cluster running in the cloud.

Attach the Virtual private gateway to VPC

In this step, we will attach the previously created private gateway to our EKS VPC. This connection allows the VPC to establish communication with the on-premises network through the Site-to-Site VPN.

Setup Routes in VPC.

To enable communication between resources in our VPC and the on-premises network, we need to update the routing tables. This configuration ensures that traffic destined for the local network is directed through the private gateway.

We will update each of the relevant route tables: private, public, and intra, by adding a route to the local CIDR block 192.168.100.0/23 with the Private Gateway as the target.

Configure the client.

Now that everything is set up on the AWS side, it’s time to configure the VPN client on your local environment.

In my case, the client is a Ubuntu 24.04 virtual machine located in a DMZ, meaning any request sent to my public IP is forwarded to this machine. We’ll be using Strongswan as the VPN client to connect to AWS’s Site-to-Site VPN.

Step 1: Download the VPN Configuration

Go to the VPC Dashboard → Site-to-Site VPN Connections, select your VPN connection, and click Download Configuration. Choose Strongswan as the vendor.

Step 2: Install Strongswan

On your VPN client machine, install Strongswan using:

apt update
apt install strongswan

Step 3: Configure the VPN Tunnels

Use the provided AWS configuration as a base. Edit the file /etc/ipsec.conf with your VPN tunnel definitions, updating values where needed (e.g., IP addresses, leftupdown hook). Here’s an example configuration:

config setup
        uniqueids = no

conn Tunnel1
 auto=start
 left=%defaultroute
 leftid=186.10.xx.xx
 right=34.194.25.197
 type=tunnel
 leftauth=psk
 rightauth=psk
 keyexchange=ikev1
 ike=aes128-sha1-modp1024
 ikelifetime=8h
 esp=aes128-sha1-modp1024
 lifetime=1h
 keyingtries=%forever
 leftsubnet=0.0.0.0/0
 rightsubnet=0.0.0.0/0
 dpddelay=10s
 dpdtimeout=30s
 dpdaction=restart
 mark=100
 leftupdown="/etc/ipsec.d/aws-updown.sh -ln Tunnel1 -ll 169.254.41.174/30 -lr 169.254.41.173/30 -m 100 -r 10.0.0.0/16"

conn Tunnel2
 auto=start
 left=%defaultroute
 leftid=186.10.xx.xx
 right=100.27.149.167
 type=tunnel
 leftauth=psk
 rightauth=psk
 keyexchange=ikev1
 ike=aes128-sha1-modp1024
 ikelifetime=8h
 esp=aes128-sha1-modp1024
 lifetime=1h
 keyingtries=%forever
 leftsubnet=0.0.0.0/0
 rightsubnet=0.0.0.0/0
 dpddelay=10s
 dpdtimeout=30s
 dpdaction=restart
 mark=200
 leftupdown="/etc/ipsec.d/aws-updown.sh -ln Tunnel2 -ll 169.254.125.226/30 -lr 169.254.125.225/30 -m 200 -r 10.0.0.0/16"

Step 4: Update the updown Script

The file /etc/ipsec.d/aws-updown.sh manages the VTI interface setup for each tunnel. If your VPN client is behind a NAT (like mine), you need to manually set the src IP in the routing section.

Locate the add_route() function and modify the ip route add line to include your local machine’s internal IP (192.168.100.100 in this case):

ip route add ${i} dev ${TUNNEL_NAME} metric ${TUNNEL_MARK}

Here’s the complete updown script:

#!/bin/bash

while [[ $# > 1 ]]; do
 case ${1} in
  -ln|--link-name)
   TUNNEL_NAME="${2}"
   TUNNEL_PHY_INTERFACE="${PLUTO_INTERFACE}"
   shift
   ;;
  -ll|--link-local)
   TUNNEL_LOCAL_ADDRESS="${2}"
   TUNNEL_LOCAL_ENDPOINT="${PLUTO_ME}"
   shift
   ;;
  -lr|--link-remote)
   TUNNEL_REMOTE_ADDRESS="${2}"
   TUNNEL_REMOTE_ENDPOINT="${PLUTO_PEER}"
   shift
   ;;
  -m|--mark)
   TUNNEL_MARK="${2}"
   shift
   ;;
  -r|--static-route)
   TUNNEL_STATIC_ROUTE="${2}"
   shift
   ;;
  *)
   echo "${0}: Unknown argument \"${1}\"" >&2
   ;;
 esac
 shift
done

command_exists() {
 type "$1" >&2 2>&2
}

create_interface() {
 ip link add ${TUNNEL_NAME} type vti local ${TUNNEL_LOCAL_ENDPOINT} remote ${TUNNEL_REMOTE_ENDPOINT} key ${TUNNEL_MARK}
 ip addr add ${TUNNEL_LOCAL_ADDRESS} remote ${TUNNEL_REMOTE_ADDRESS} dev ${TUNNEL_NAME}
 ip link set ${TUNNEL_NAME} up mtu 1419
}

configure_sysctl() {
 sysctl -w net.ipv4.ip_forward=1
 sysctl -w net.ipv4.conf.${TUNNEL_NAME}.rp_filter=2
 sysctl -w net.ipv4.conf.${TUNNEL_NAME}.disable_policy=1
 sysctl -w net.ipv4.conf.${TUNNEL_PHY_INTERFACE}.disable_xfrm=1
 sysctl -w net.ipv4.conf.${TUNNEL_PHY_INTERFACE}.disable_policy=1
}

add_route() {
 IFS=',' read -ra route <<< "${TUNNEL_STATIC_ROUTE}"
     for i in "${route[@]}"; do
     ip route add ${i} dev ${TUNNEL_NAME} metric ${TUNNEL_MARK} src 192.168.100.100
 done
 iptables -t mangle -A FORWARD -o ${TUNNEL_NAME} -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
 iptables -t mangle -A INPUT -p esp -s ${TUNNEL_REMOTE_ENDPOINT} -d ${TUNNEL_LOCAL_ENDPOINT} -j MARK --set-xmark ${TUNNEL_MARK}
 ip route flush table 220
}

cleanup() {
        IFS=',' read -ra route <<< "${TUNNEL_STATIC_ROUTE}"
        for i in "${route[@]}"; do
            ip route del ${i} dev ${TUNNEL_NAME} metric ${TUNNEL_MARK}
        done
 iptables -t mangle -D FORWARD -o ${TUNNEL_NAME} -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
 iptables -t mangle -D INPUT -p esp -s ${TUNNEL_REMOTE_ENDPOINT} -d ${TUNNEL_LOCAL_ENDPOINT} -j MARK --set-xmark ${TUNNEL_MARK}
 ip route flush cache
}

delete_interface() {
 ip link set ${TUNNEL_NAME} down
 ip link del ${TUNNEL_NAME}
}

# main execution starts here

command_exists ip || echo "ERROR: ip command is required to execute the script, check if you are running as root, mostly to do with path, /sbin/" >&2 2>&2
command_exists iptables || echo "ERROR: iptables command is required to execute the script, check if you are running as root, mostly to do with path, /sbin/" >&2 2>&2
command_exists sysctl || echo "ERROR: sysctl command is required to execute the script, check if you are running as root, mostly to do with path, /sbin/" >&2 2>&2

case "${PLUTO_VERB}" in
 up-client)
  create_interface
  configure_sysctl
  add_route
  ;;
 down-client)
  cleanup
  delete_interface
  ;;
esac

Step 5: Add Shared Secrets

Open /etc/ipsec.secrets and add the shared secrets provided in the configuration file:

# This file holds shared secrets or RSA private keys for authentication.

# RSA private key for this host, authenticating it to any other host
# which knows the public part.

186.10.xx.xx 34.194.25.197 : PSK "kPcU.tJ_sA33J7Z.I4f4gxxxxx"
186.10.xx.xx 100.27.149.167 : PSK "YtaQSGhvMKV4aLQx.4wxxxxxxx"

Step 6: Start the VPN Service

Restart the service and check the tunnel status:

systemctl restart ipsec
ipsec status

The following video demonstrates the complete setup step by step:

If everything is correctly configured, both tunnels should show as “up” in the AWS Console → Site-to-Site VPN Connections page.

At this stage, the network connectivity between the EKS cluster and the on-premises environment is fully configured:

In Part 2, we will provision the EKS cluster using OpenTofu, set up the SSM activator, and register the on-premises nodes using nodeadm with the Cilium CNI driver.

In Part 3, we will install the AWS Load Balancer Controller, configure NGINX Ingress, and deploy our application across both EC2 and hybrid (on-prem) nodes.