DEV Community: Pendela BhargavaSai

K3s vs Kubernetes: A Deep Dive into Control Plane Architecture

Pendela BhargavaSai — Tue, 09 Jun 2026 03:30:00 +0000

Not just "what's different" — but WHY it's different, HOW each component works under the hood, and WHEN to choose which.

🧠 Why This Post Exists

Every "K3s vs K8s" article you've read probably gave you a table with checkmarks and said "K3s is lightweight." That's true — but why is it lightweight? What did Rancher actually strip out, merge, or replace? What are the architectural trade-offs you inherit when you deploy K3s in production?

This post tears open both control planes component by component. We'll go deep into what each piece actually does at the byte level, then see how K3s reimagines it.

🏗️ The Kubernetes Control Plane: A Ground-Up Look

Before comparing, let's build a mental model of each standard Kubernetes control plane component. Not the 30-second version — the real one.

1. 🔵 kube-apiserver — The Brain's Frontal Lobe

What It Actually Does

The API server is not just a REST endpoint. It is the only component in Kubernetes that talks directly to etcd. Every other component — scheduler, controller-manager, kubelet — communicates exclusively through the API server. This is a deliberate architectural decision called the hub-and-spoke pattern.

When you run kubectl apply -f deployment.yaml, here's what actually happens:

kubectl → HTTPS → kube-apiserver
                    │
                    ├── 1. Authentication (Who are you?)
                    │       └── x509 certs / Bearer tokens / OIDC /Webhook
                    │
                    ├── 2. Authorization (Can you do this?)
                    │       └── RBAC / ABAC / Node / Webhook evaluators
                    │
                    ├── 3. Admission Control (Should this be allowed?)
                    │       ├── Mutating Webhooks  ← can MODIFY the object
                    │       └── Validating Webhooks ← can REJECT theobject
                    │
                    ├── 4. Schema Validation
                    │       └── OpenAPI v3 schema enforcement per GVK
                    │
                    └── 5. Persist to etcd
                            └── /registry/deployments/default/my-app

The Watch Mechanism — The Heartbeat of Kubernetes

The API server implements a long-poll watch mechanism over HTTP/2. This is what makes Kubernetes reactive rather than polling-based.

# You can see this yourself
kubectl get pods --watch -v=9
# Watch the raw HTTP stream — it's a chunked HTTP response that stays open

Every controller, scheduler, and kubelet maintains a persistent informer — a cached watch stream from the API server. The informer pattern:

Does an initial LIST to populate local cache
Starts a WATCH from the resource version of that LIST
On disconnect, re-watches from the last known resourceVersion
The API server buffers events in a watchCache in memory (configurable with --watch-cache-sizes)

                    ┌─────────────────────────┐
                    │      kube-apiserver     │
                    │                         │
                    │  ┌─────────────────┐    │
                    │  │   etcd watch    │    │
                    │  └────────┬────────┘    │
                    │           │             │
                    │  ┌────────▼────────┐    │
                    │  │   watchCache    │    │  ← In-memory ring buffer
                    │  └────────┬────────┘    │
                    │           │             │
                    └───────────┼─────────────┘
                                │
              ┌─────────────────┼──────────────────┐
              │                 │                  │
         ┌────▼────┐      ┌─────▼─────┐     ┌─────▼─────┐
         │Scheduler│      │Controller │     │  kubelet  │
         │Informer │      │  Informer │     │  Informer │
         └─────────┘      └───────────┘     └───────────┘

Aggregation Layer & CRDs

The API server can extend itself via two mechanisms:

CRDs (Custom Resource Definitions): Schema is stored in etcd, handled natively by the API server itself
Aggregation Layer (AA): Proxy traffic to an external API server (used by metrics-server, KEDA, etc.)

# CRD — API server owns the storage
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.example.com

# AA — API server proxies to external server
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system

Production Tuning Knobs

kube-apiserver \
  --max-requests-inflight=400 \          # Max non-mutating concurrent requests
  --max-mutating-requests-inflight=200 \ # Max mutating concurrent requests
  --watch-cache-sizes=pods#1000 \        # Per-resource watch cache sizes
  --enable-admission-plugins=NodeRestriction,PodSecurity \
  --audit-log-path=/var/log/audit.log \
  --audit-policy-file=/etc/k8s/audit-policy.yaml

2. 🟣 etcd — The Distributed Brain's Memory

What etcd Actually Is

etcd is a distributed key-value store built on the Raft consensus algorithm. It's not a database in the traditional sense — it's a fault-tolerant state machine where every write must be agreed upon by a quorum of nodes before it's committed.

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   etcd-0    │     │   etcd-1    │     │   etcd-2    │
│  (LEADER)   │◄────│  (FOLLOWER) │     │  (FOLLOWER) │
│             │────►│             │     │             │
└──────┬──────┘     └─────────────┘     └──────▲──────┘
       │                                       │
       └───────────────────────────────────────┘
                    Raft Heartbeats

Raft in Plain English

Leader Election: One node becomes leader. It sends heartbeats. If 2+ nodes don't hear a heartbeat, they call an election.
Log Replication: Every write goes to the leader. Leader appends it to its log and replicates it to followers. Once a majority acknowledges, the write is committed.
Quorum Math: (n/2) + 1 nodes must agree. For 3 nodes: 2. For 5 nodes: 3.

etcd write path:
Client → Leader APPEND entry to log
         Leader SEND AppendEntries RPC to all followers
         Followers ACKNOWLEDGE
         Leader COMMITS when the majority ack
         Leader RESPONDS to client
         Leader NOTIFIES followers of the commit

How Kubernetes Data Lives in etcd

All Kubernetes objects are stored under /registry/ with the structure:

/registry/{resource-type}/{namespace}/{name}

Examples:
/registry/pods/default/nginx-7d8b9f-xyz
/registry/deployments/kube-system/coredns
/registry/secrets/default/my-secret
/registry/events/default/pod-scheduled-event

The data is serialized using protobuf (not JSON!) for efficiency. You can inspect it:

# Decode an etcd value
etcdctl get /registry/pods/default/nginx \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  | auger decode  # github.com/jpbetz/auger

MVCC — Multi-Version Concurrency Control

etcd uses MVCC, meaning it keeps multiple historical versions of every key. Each write increments a global revision counter. The API server uses this resourceVersion for watch ordering and conflict detection.

# See the revision
etcdctl get /registry/pods/default/nginx -w json | jq .header.revision

When etcd's keyspace grows too large (default compaction at 2GB), older revisions are compacted — deleted. This is why very old watches can fail with "compacted" errors.

etcd Failure Modes You Must Know

Scenario	What Happens
1 node fails (3-node cluster)	Cluster continues. Writes still work.
2 nodes fail (3-node cluster)	CLUSTER STOPS ACCEPTING WRITES. API server returns 503.
Leader fails	Election happens. ~150-300ms downtime while new leader is elected.
Network partition	Minority partition goes read-only. Majority continues.
etcd OOM	API server loses state store. Catastrophic.

⚠️ This is the critical difference with K3s. If you're running K3s with embedded SQLite, you get zero HA for the datastore by default.

3. 🟡 kube-scheduler — The CPU-Time Auctioneer

What It Actually Does

The scheduler watches for Pods in Pending state (no nodeName assigned) and decides which Node they should run on. It does NOT place the pod — it simply writes the chosen nodeName to the Pod spec in etcd via the API server. The kubelet on that node then sees its name and starts the pod.

Pod created (nodeName: "")  →  Scheduler sees it via watch
                            →  Runs filtering + scoring
                            →  Writes nodeName to Pod
                            →  kubelet on that node sees the Pod
                            →  kubelet pulls image + starts container

The Scheduling Framework — Two-Phase Deep Dive

Scheduling happens in two phases: Filtering and Scoring.

Phase 1: Filtering (Hard Constraints — binary pass/fail)

All Nodes
    │
    ▼
┌─────────────────────────────────────────────────────┐
│  Filter Plugins (run in parallel, any fail = remove) │
│                                                     │
│  • NodeUnschedulable  — node.spec.unschedulable?    │
│  • NodeAffinity       — matchLabels on node?        │
│  • TaintToleration    — pod tolerates node taints?  │
│  • PodTopologySpread  — spread constraints met?     │
│  • VolumeBinding      — PVC can bind to this node?  │
│  • NodeResourcesFit   — enough CPU/mem/GPU?         │
│  • NodePorts          — hostPort conflicts?         │
└─────────────────────────────────────────────────────┘
    │
    ▼
Feasible Nodes (subset)

Phase 2: Scoring (Soft Preferences — 0-100 score)

Feasible Nodes
    │
    ▼
┌─────────────────────────────────────────────────────┐
│  Score Plugins (weighted sum)                        │
│                                                     │
│  • LeastAllocated       — prefer less loaded nodes  │
│  • NodeAffinity         — preferred affinities      │
│  • InterPodAffinity     — co-locate or spread pods  │
│  • ImageLocality        — prefer nodes with image   │
│  • TaintToleration      — fewer preferred taints    │
│  • TopologySpreadConstraint — balance spread        │
└─────────────────────────────────────────────────────┘
    │
    ▼
Highest Score Node → Binding (nodeName written)

Preemption — What Happens When No Node Passes Filtering

If no node can fit the Pod, the scheduler checks if lower priority pods can be evicted to make room:

Find nodes where evicting lower-priority pods creates enough room
Pick the node that requires evicting the fewest/lowest-priority pods
Send eviction requests → evicted pods are deleted → pending pod is scheduled

# Priority classes matter here
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
---
# System-critical pods have value: 2000001000
# They will preempt your workloads if nodes are tight

The Binding Cache — Optimistic Concurrency

The scheduler maintains an assumed pod cache. After scoring but before the API server confirms the bind, the scheduler optimistically assumes the pod is placed and accounts for that node's capacity. This prevents scheduling thrash in high-throughput clusters.

4. 🟢 kube-controller-manager — The Reconciliation Engine

What It Actually Is

The controller manager is a single binary that runs ~30+ independent control loops as goroutines. Each controller watches specific resource types and reconciles desired state vs actual state.

# The reconciliation loop in pseudocode (every controller)
for {
    desired := get_desired_state_from_api_server()
    actual  := get_actual_state_from_world()

    if desired != actual {
        take_action_to_make_actual_match_desired()
    }

    sleep(resync_period)  // default: 10min
}

Key Controllers and What They Actually Do

ReplicaSet Controller

Watches: ReplicaSets, Pods
Loop:
  current_pods = list pods with matching selector
  delta = replicaset.spec.replicas - len(current_pods)
  if delta > 0: create `delta` pods
  if delta < 0: delete abs(delta) pods (by priority: unscheduled first)

Deployment Controller

Watches: Deployments, ReplicaSets
Loop:
  desired_rs = compute_hash(deployment.spec.template)
  if no RS with that hash: create new RS
  scale up new RS, scale down old RS (by strategy: RollingUpdate or Recreate)
  update deployment.status (readyReplicas, conditions, etc.)

Node Controller — This one is critical to understand

Watches: Nodes
Loop:
  for each node:
    if no heartbeat for node-monitor-grace-period (default 40s):
      set NodeReady=Unknown
    if no heartbeat for pod-eviction-timeout (default 5min):
      taint node with node.kubernetes.io/unreachable:NoExecute
      (this triggers pod eviction by the taint manager)

EndpointSlice Controller — How Services actually work

Watches: Services, Pods
Loop:
  for each service:
    pods = list pods matching service.spec.selector where pod.status.ready=true
    build EndpointSlices (groups of 100 endpoints each)
    write EndpointSlices to API server
    (kube-proxy watches EndpointSlices and updates iptables/ipvs rules)

Informer + WorkQueue Architecture

Every controller is built on the same pattern:

API Server Watch
      │
      ▼
   Informer
   (local cache)
      │
      ▼  (on change event)
  WorkQueue  ←──── rate-limited, deduplicated
      │
      ▼
  Worker goroutines (usually 1-5)
      │
      ▼
  Reconcile function
      │
      ├── Success → remove from queue
      └── Failure → re-queue with exponential backoff

This pattern means controllers are eventually consistent — they don't act on every single event, they converge to the desired state over time.

5. 🔴 cloud-controller-manager — The Cloud API Bridge

What It Actually Does

The CCM was extracted from kube-controller-manager in Kubernetes 1.11 specifically to decouple Kubernetes from cloud provider APIs. It runs cloud-specific control loops:

Node Controller (cloud variant)

On new Node joining:
  1. Fetch instance metadata from cloud API
     (AWS EC2 DescribeInstances / GCP ComputeInstances)
  2. Apply cloud provider labels:
     - topology.kubernetes.io/zone = us-east-1a
     - node.kubernetes.io/instance-type = m5.xlarge
  3. Set node addresses (internal/external IP from cloud metadata)
  4. Check if instance still exists periodically
     → If terminated in cloud: delete the Node object

Route Controller (AWS/GCP specific)

For each node:
  ensure cloud routing table has route:
  pod-cidr (e.g., 10.244.1.0/24) → node instance-id

This is how pod-to-pod routing works across nodes
WITHOUT an overlay network on supported clouds

Service Controller — The LoadBalancer Magic

Watch Services with type=LoadBalancer:
  on CREATE: call cloud API → create load balancer
             update service.status.loadBalancer.ingress with external IP
  on UPDATE: update LB listener rules / health checks
  on DELETE: delete cloud load balancer

This is why kubectl get svc shows <pending> for LoadBalancer services until the cloud LB is provisioned.

⚡ The K3s Control Plane: Architectural Reimagination

Now let's look at what K3s does differently — not just "it's smaller" but architecturally why.

K3s Single Binary Philosophy

K3s ships as a single ~70MB binary (k3s) that embeds:

k3s binary
├── k3s-server (control plane)
│   ├── kube-apiserver
│   ├── kube-controller-manager
│   ├── kube-scheduler
│   ├── kubelet
│   ├── kube-proxy
│   ├── embedded containerd
│   ├── embedded CoreDNS
│   ├── embedded Flannel (CNI)
│   ├── embedded Traefik (ingress)
│   ├── embedded ServiceLB (load balancer)
│   └── embedded local-path-provisioner (storage)
└── k3s-agent (worker)
    ├── kubelet
    ├── kube-proxy
    └── embedded containerd

This is not containerized — these are linked as Go packages into a single binary. Startup goes from ~3 minutes (typical K8s) to under 30 seconds.

1. K3s API Server — Same Core, Slimmer Defaults

The K3s API server is still the upstream kube-apiserver — but K3s wraps it with:

Removed/Disabled by Default:

Alpha feature gates are disabled
Cloud provider plugins: --cloud-provider=external not set (no CCM)
Several admission plugins that assume cloud infra

The K3s Tunnel Proxy — Replacing the CCM Node Controller

K3s introduces a reverse tunnel from agent → server. In standard K8s, the API server connects to the kubelet for exec/logs/port-forward. In K3s:

Standard K8s:
  kube-apiserver → kubelet:10250  (API server initiates)
  Requires API server to reach all nodes directly

K3s:
  k3s-agent → k3s-server:6443 (agent initiates)
  ┌────────────────────────────────────────────────┐
  │  WebSocket tunnel maintained by agent          │
  │  All kubelet traffic flows THROUGH this tunnel │
  └────────────────────────────────────────────────┘

This is why K3s works behind NAT without special networking — agents reach out, not the server. This is a fundamental architectural shift that enables edge/IoT deployments.

2. SQLite / Kine — The etcd Abstraction Layer

This is the most significant architectural difference.

K3s introduces Kine (Kubernetes Is Not Etcd) — a shim that translates etcd's gRPC API into SQL queries.

kube-apiserver
      │
      │  etcd gRPC v3 protocol (ListWatch, Txn, etc.)
      ▼
┌──────────────┐
│     Kine     │  ← translation layer
│  (etcd shim) │
└──────┬───────┘
       │  SQL queries
       ▼
┌──────────────┐
│   SQLite /   │  ← actual datastore
│  PostgreSQL  │
│    MySQL     │
│   DQLite     │
└──────────────┘

How Kine Implements the etcd Watch API:

etcd's watch is event-driven via gRPC streams. SQL databases don't natively support this. Kine implements it via:

-- Kine's core table
CREATE TABLE kine (
  id      INTEGER PRIMARY KEY AUTOINCREMENT,  -- acts as etcd revision
  name    TEXT,                               -- the key (/registry/pods/...)
  created INTEGER,
  deleted INTEGER,
  create_revision INTEGER,
  prev_revision   INTEGER,
  lease   INTEGER,
  value   BLOB,                              -- the protobuf-encoded object
  old_value BLOB
);

-- Watch is implemented as polling:
-- SELECT * FROM kine WHERE id > last_seen_id ORDER BY id
-- Run every ~100ms — NOT event-driven like real etcd

The Implications:

For small clusters: unnoticeable
For large clusters: polling adds latency to watch events
SQLite: single-writer, no HA (single node only)
PostgreSQL/MySQL with Kine: HA possible but watch latency higher than etcd

DQLite — Embedded Distributed SQLite (Experimental)

For HA without an external DB, K3s can use DQLite — a distributed SQLite implementation using Raft (similar to etcd but built on SQLite). It's embedded in the binary and doesn't require an external DB.

# K3s with embedded HA using DQLite
k3s server --cluster-init   # First server (bootstrap)
k3s server --server https://first-server:6443 --token <token>  # Join as HA peer

3. K3s Controller Manager — Pruned and Extended

K3s runs the upstream kube-controller-manager with several modifications:

Removed Controllers:

cloud-node controller (no cloud metadata fetching)
cloud-node-lifecycle controller
route controller (no cloud routes)
service controller (replaced by ServiceLB)

Added: ServiceLB (a.k.a. Klipper LoadBalancer)

Instead of calling a cloud API to provision a load balancer, K3s runs a DaemonSet-based solution:

Service type=LoadBalancer created
        │
        ▼
ServiceLB Controller watches for it
        │
        ▼
Creates a DaemonSet:
  - Runs a pod on every node with hostPort matching service ports
  - The pod does iptables DNAT → service ClusterIP
        │
        ▼
Every node's IP becomes a valid entry point
(no external LB needed)

# What ServiceLB actually deploys under the hood
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: svclb-my-service
  namespace: kube-system
spec:
  template:
    spec:
      hostNetwork: true
      containers:
      - name: lb-port-80
        image: rancher/klipper-lb:latest
        ports:
        - hostPort: 80       # binds on every node
          containerPort: 80
        env:
        - name: SRC_PORT
          value: "80"
        - name: DEST_PROTO
          value: TCP
        - name: DEST_IP
          value: "10.96.100.50"  # ClusterIP
        - name: DEST_PORT
          value: "80"

4. K3s Scheduler — Unchanged but Co-located

The scheduler in K3s is the unmodified upstream kube-scheduler. However, it runs as a goroutine inside the k3s-server binary rather than as a separate process.

The key difference is operational:

In K8s: scheduler can be independently scaled, upgraded, or replaced (e.g., with Volcano, Yunikorn)
In K3s: scheduler is embedded — replacing it requires rebuilding or running an external scheduler with leader election disabled on the built-in one

5. The Flannel CNI — Embedded Networking

Standard K8s requires you to install a CNI (Calico, Cilium, Flannel, Weave) separately. K3s embeds Flannel with VXLAN as the default backend.

Pod on Node 1 (10.42.1.5) → Pod on Node 2 (10.42.2.7)

Standard K8s + Calico:
  10.42.1.5 → BGP route → 10.42.2.7 (no encapsulation on supported networks)

K3s + Flannel VXLAN:
  10.42.1.5 → VXLAN encapsulate → eth0:8472 → Node 2 → decapsulate → 10.42.2.7
  (works everywhere, slight overhead from encapsulation)

K3s also supports swapping Flannel for Cilium or Calico if you disable the built-in:

k3s server --flannel-backend=none --disable-network-policy
# Then install Cilium/Calico manually

📊 Side-by-Side Deep Comparison

Dimension	Standard Kubernetes	K3s
Deployment model	Separate processes (+ etcd cluster)	Single binary, all-in-one
API server	Full upstream, all features	Full upstream, conservative defaults
Datastore	etcd (Raft, event-driven watch)	SQLite/Kine (SQL polling) or embedded DQLite
Watch latency	~10ms (event-driven)	~100ms (polling on SQL backends)
HA datastore	etcd cluster (3/5 nodes)	External DB + Kine OR embedded DQLite
Control plane HA	Multiple API server replicas	Multiple k3s-server nodes possible
Cloud integration	cloud-controller-manager	No CCM, uses ServiceLB + node-ip flags
LoadBalancer	Cloud LB (AWS ELB, GCP GLB)	ServiceLB DaemonSet (hostPort)
Ingress	Bring your own (nginx, traefik)	Traefik v2 embedded
CNI	Bring your own	Flannel (VXLAN) embedded
DNS	Bring your own CoreDNS	CoreDNS embedded
Storage	Bring your own CSI	local-path-provisioner embedded
Kubelet location	Separate binary on worker	Embedded in k3s binary
API server → kubelet	Direct connection (port 10250)	Reverse WebSocket tunnel
Memory (control plane)	~2GB+ (separate processes)	~512MB (single process)
Startup time	2-5 minutes	20-30 seconds
Alpha feature gates	Available	Disabled by default
Admission webhooks	Full support	Full support
CRDs	Full support	Full support
RBAC	Full support	Full support
Audit logging	Configurable	Configurable
Scheduler extensibility	Scheduler profiles, plugins	Embedded; replace with external
Controller extensibility	Separate binary, hot-swap	Embedded goroutine
Upgrades	Independent component upgrades	Single binary upgrade
Edge/NAT traversal	Requires direct reachability	Native via reverse tunnel
ARM support	Separate builds	Native multi-arch in single release

🔑 When to Choose What

Choose Standard Kubernetes When:

✅ 100+ node clusters
✅ Financial / regulated workloads requiring etcd for compliance
✅ You need independent control plane component upgrades
✅ You're using cloud-managed control planes (EKS, GKE, AKS)
✅ You need custom scheduler profiles (ML batch, GPU scheduling)
✅ Multi-tenancy with strong isolation requirements
✅ You need external etcd for ultra-high availability
✅ Team has K8s expertise and infra budget

Choose K3s When:

✅ Edge computing (retail, industrial, remote sites)
✅ IoT / ARM devices (Raspberry Pi clusters)
✅ CI/CD ephemeral clusters (fast startup is critical)
✅ Development environments (minimal resource usage)
✅ Single-node homelab or small on-prem clusters
✅ Clusters behind NAT (reverse tunnel is a killer feature)
✅ Teams that want "it just works" with less Ops overhead
✅ Bare metal without a cloud provider
✅ Air-gapped environments (single binary, easy to ship)

🔭 The Architecture Decision Tree

Do you need >50 nodes?
├── YES → Standard K8s (EKS/GKE/AKS or kubeadm)
└── NO
    ├── Are you on edge/IoT/ARM?
    │   └── YES → K3s (purpose-built for this)
    ├── Do you need cloud LoadBalancer integration?
    │   └── YES → Standard K8s with CCM
    ├── Is startup speed critical? (CI/CD, dev envs)
    │   └── YES → K3s
    ├── Do you need etcd for compliance/audit?
    │   └── YES → Standard K8s
    └── Default recommendation for <20 nodes on-prem?
        └── K3s (less to manage, same K8s API)

🎯 Closing Thoughts

K3s isn't "Kubernetes with stuff removed." It's a purpose-built reimagining of the control plane for constrained environments. Rancher made deliberate trade-offs:

etcd → Kine/SQLite: Sacrificed watch latency and native HA for operational simplicity
Separate binaries → Single binary: Sacrificed independent upgradeability for atomic deployments
CCM → ServiceLB: Sacrificed cloud-native LB for zero-dependency load balancing
Direct kubelet access → Reverse tunnel: Sacrificed simplicity for NAT traversal capability

The result is a distribution that runs the full Kubernetes API on a Raspberry Pi with 512MB of RAM, starts in 30 seconds, and works behind NAT — things standard K8s simply wasn't designed for.

Both are Kubernetes. Both run your workloads. The control plane is where the real difference lives.

I Broke My Proxmox Home Lab with a GPU Passthrough - Here's How I Fixed It

Pendela BhargavaSai — Tue, 19 May 2026 03:30:00 +0000

How a Kubernetes worker VM with a passed-through AMD GPU sent my entire home lab into an infinite crash loop — and the GRUB-level trick that saved it.

What I Was Trying to Do

My home lab runs on a Mini PC with Proxmox as the hypervisor. The setup hosts a mix of LXC containers and KVM virtual machines — Jellyfin for media, a Pi-hole, a Kubernetes cluster (k3s), Cloudflare tunnels, and a bunch of other self-hosted services.

I was upgrading the hardware configuration of VM 104 (k3s-vm-worker) — one of the worker nodes in my k3s Kubernetes cluster. The goal was straightforward: pass through the host's AMD GPU directly into the VM so the Kubernetes worker could handle GPU-accelerated workloads.

In Proxmox, GPU passthrough (PCIe passthrough) requires a few things:

IOMMU enabled in BIOS (AMD-Vi for AMD platforms)
amd_iommu=on iommu=pt kernel parameters in GRUB
Machine type set to q35 (required for PCIe passthrough)
The PCI device was added as hostpci0 in the VM's hardware config

I configured all of this. The hardware tab showed the PCI device (0000:e6:00, highlighted in orange — a warning sign I probably should have paid more attention to). I saved the config, felt good about it, and rebooted.

That was my mistake.

What Happened — The Crash Loop

The Mini PC came back up, Proxmox started loading, and then — nothing. The host became completely unresponsive. No web UI. No SSH. Nothing.

I power-cycled it. Same thing. It would power on, start booting, and crash before I could even get to the Proxmox web interface.

Here's what was actually happening, which I only understood after I could finally get in and check the logs:

VM 104 had "Start at boot" set to Yes.

The moment Proxmox finished loading, it triggered a "Bulk Start VMs" task — its automatic process for starting all VMs flagged to auto-start. VM 104 was in that queue. The second that task reached VM 104, Proxmox tried to initialise the PCIe passthrough, the VM grabbed the GPU from the host, and the entire system crashed — hard.

I panicked and quickly got back into the web UI during a narrow window and changed "Start at boot" to No. Thought that would fix it.

It didn't.

Proxmox had already queued the bulk start task the moment it turned on. Changing the setting mid-flight did nothing. The next boot, same crash. The boot after that, same crash.

I was locked in an auto-start crash loop with no way out through the web UI.

Understanding the Problem — Why It Was So Hard to Escape

This is the part that makes this specific failure mode so nasty:

Boot → Proxmox loads → "Bulk Start VMs" fires instantly →
VM 104 grabs GPU → Host kernel panics → System crashes →
Reboot → repeat forever

The window between "Proxmox is up enough to serve the web UI" and "VM 104 has already been queued and grabbed the GPU" was too small to intervene through normal means. Every time I managed to get in and change something, the crash had already been scheduled.

The only way out was to break the loop before Proxmox had any chance to load its VM management services at all — which means intervening at the GRUB level, before the operating system even finishes booting.

The Solution — Masking the VM Start Service at Boot

This required plugging a physical keyboard and monitor into the Mini PC. No SSH, no remote — physical access only.

Step 1: Stop the autoboot timer at GRUB

Power on the machine and watch the screen. The moment the blue Proxmox GRUB menu appears, press the Down Arrow key immediately. This stops the autoboot countdown timer and lets you stay at the GRUB menu.

Step 2: Edit the boot command

Highlight the top "Proxmox VE" entry and press e to edit the boot command. You'll see the full kernel command line. Navigate to the end of the linux line.

Remove amd_iommu=on iommu=pt (these were the IOMMU parameters I'd added earlier that enabled the passthrough).

Then, at the very end of the linux line, add:

systemd.mask=pve-guests.service

So the line ends with:

ro quiet systemd.mask=pve-guests.service

Press Ctrl + X to boot with this modified command.

What `systemd.mask=pve-guests.service` actually does

pve-guests.service is the systemd unit responsible for the "Bulk Start VMs" behaviour in Proxmox. Masking it via the kernel command line at boot tells systemd to completely skip that service for this boot session only — it doesn't persist across reboots.

The result:

✅ Proxmox boots normally
✅ Web UI comes up
✅ Network is available
✅ Zero VMs start automatically
✅ Host does not crash

Step 3: Fix the VM config via web UI

Once safely in the web UI with no VMs running:

VM 104 → Options → "Start at boot" → No
VM 104 → Hardware → PCI Device (hostpci0) → Remove
VM 104 → Hardware → Machine → change from q35 back to i440fx
Reboot normally — don't touch the GRUB menu this time

Plot Twist — The GPU Renamed Itself

After fixing the crash loop and getting the host stable, I started my LXC containers — CT 204 (jellyfin-arr) and CT 208 (linkwarden) — and hit a completely different error:

Error: Device /dev/dri/card0 ...

Both containers refused to start. I checked the DRI devices on the host:

ls -la /dev/dri/

Output:

crw-rw---- 1 root video  226,   1 May 13 13:51 card1
crw-rw---- 1 root render 226, 128 May 13 13:51 renderD128

The AMD GPU loaded perfectly (Kernel driver in use: amdgpu). But Linux had decided to name it card1 instead of card0.

This is a surprisingly common Linux behaviour: when you change BIOS IOMMU settings, or reboot without an HDMI cable connected to a GPU, the kernel can enumerate DRM devices in a different order and assign a different cardN number. The GPU is fine. The driver is fine. The device node just got a different name.

My LXC containers were hardcoded to pass through /dev/dri/card0. The fix was trivial once I understood it:

CT 204 → Resources → Device (dev1) — change /dev/dri/card0 to /dev/dri/card1
CT 208 → Resources → Device (dev1) — same change

Both containers started instantly. Hardware transcoding in Jellyfin came back up immediately.

Root Cause Analysis

Looking back, there were actually three separate issues stacked on top of each other:

#	Issue	Root Cause
1	Host crash loop	VM with PCIe passthrough set to "Start at boot" — GPU grabbed before host could stabilise
2	Can't escape via web UI	`pve-guests.service` fires before web UI is interactive enough to intervene
3	LXC containers broken	GPU renamed from `card0` to `card1` after IOMMU BIOS change

The orange 0000:e6:00 text in the VM Hardware tab was the visual warning I missed — in Proxmox, orange on a PCI device entry typically indicates the device may not be properly isolated in its IOMMU group, or that there's a configuration issue worth investigating before saving and booting.

Lessons Learned

1. Never enable "Start at boot" on a VM with PCIe passthrough until you've verified the passthrough works correctly in a manual start first.

Test the VM with a manual start. Confirm the host stays stable. Confirm the VM boots correctly with the passed-through device. Only then enable auto-start.

2. systemd.mask=<service> at GRUB is your emergency brake.

Any time Proxmox is in a state where it crashes before you can intervene through the web UI, this technique gives you a clean boot with surgical control over what services start. pve-guests.service is the specific one to mask for VM auto-start loops.

3. GPU device node names are not stable across reboots in Linux.

/dev/dri/card0 is not guaranteed to be your GPU after every reboot, especially when BIOS settings change. Instead of hardcoding card0, consider:

Using udev rules to create a stable symlink to your specific GPU by PCI ID
Or checking ls /dev/dri/ after any BIOS/IOMMU change before starting dependent containers

# Find which card corresponds to your GPU by PCI ID
ls -la /dev/dri/by-path/
# or
lspci -k | grep -A 3 VGA

4. PCIe passthrough machine type matters — and q35 has implications.

q35 is required for proper PCIe passthrough but behaves differently from i440fx in several ways. Switching machine types mid-configuration without a working baseline is risky. Start with a known-good VM on i440fx, verify it's stable, then migrate to q35 and add the PCI device.

5. Physical access to your home lab server is non-negotiable.

If you're running a home lab with GPU passthrough or any kind of hardware-level config, have a keyboard and monitor you can plug in. SSH and the web UI are great until they're not. The GRUB console saved this entire setup.

The Final State

After all of this:

✅ Proxmox host boots cleanly and stays stable
✅ All LXC containers (jellyfin-arr, linkwarden) running with GPU access on card1
✅ k3s Kubernetes cluster (VMs 103, 104, 105) running normally
✅ All other services (Plex, Pi-hole, Cloudflare, Pulse, Prowlarr, Pendela) unaffected
⚠️ GPU passthrough into k3s-vm-worker: parked for now, to be re-approached correctly

The GPU passthrough into the Kubernetes worker is still on the roadmap — I'll approach it differently: test in a throwaway VM first, verify IOMMU grouping, confirm stability with manual start, and only then wire it into the k3s node.

Quick Reference — Commands Used

# Check GPU device assignment on host
ls -la /dev/dri/
lspci -k | grep -A 3 -i vga

# Check IOMMU groups (run as root on Proxmox host)
for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf 'IOMMU Group %s ' "$n"
  lspci -nns "${d##*/}"
done

# GRUB emergency boot (type this at the end of linux line in GRUB edit)
systemd.mask=pve-guests.service

# Find stable GPU device path by PCI address
ls -la /dev/dri/by-path/

Running a home lab means breaking things and learning from them. This one taught me more about Proxmox internals, systemd masking, and Linux DRM device enumeration than any documentation would have. Hope it saves someone else the same panic.

Feel free to drop questions in the comments — happy to help if you're stuck in a similar loop.

The Ultimate Guide to Kubernetes Load Balancers in 2026 (K3s Edition)

Pendela BhargavaSai — Fri, 15 May 2026 03:30:00 +0000

TL;DR — Running K3s on bare metal or edge? This guide dissects every major Kubernetes load balancer — NGINX, Traefik, MetalLB, HAProxy, Envoy, Cilium, Istio, Linkerd, and K3s's own Klipper — across architecture, performance, K3s compatibility, and real-world use cases. Pick the right one for your stack, once and for all.

🧭 Why This Guide Exists

Kubernetes load balancers are one of the most confusing corners of the cloud-native ecosystem. Search for "best Kubernetes load balancer" and you'll find a dozen blog posts each recommending something different, often without context. When you throw K3s — the lightweight, single-binary Kubernetes distribution from Rancher — into the mix, the confusion compounds further.

K3s ships with its own built-in load balancer (Klipper/ServiceLB) and its own ingress controller (Traefik). But is that the right choice for your production workload? What if you need BGP routing, service mesh capabilities, or sub-millisecond latency?

This guide covers every serious option in the market today, with real benchmarks, architecture diagrams, and clear K3s-specific guidance.

🗺️ The Landscape: What Are We Even Comparing?

Before diving in, let's clarify the terminology. "Load balancer" in Kubernetes refers to multiple layers:

Layer	What It Does	Example Tools
L4 LoadBalancer (IP/TCP)	Assigns external IPs to Services	MetalLB, Klipper, Kube-VIP
L7 Ingress Controller	Routes HTTP/HTTPS traffic by host/path	NGINX, Traefik, HAProxy
Reverse Proxy / Edge Proxy	Advanced traffic shaping, retries, circuit breaking	Envoy, HAProxy
Service Mesh	East-west (pod-to-pod) traffic management + security	Istio, Linkerd, Cilium

Most real deployments combine tools from multiple layers. For K3s, a typical production stack might be: MetalLB (L4) + Traefik (L7 Ingress) + optionally Linkerd (mesh).

🔬 Competitor Deep-Dive

1. 🏠 Klipper ServiceLB (K3s Built-In)

What it is: K3s's embedded load balancer, enabled by default. Uses host ports and iptables rules to forward traffic.

Architecture:

External Traffic
      │
      ▼
[Node HostPort] ──iptables──► [ClusterIP] ──► [Pod]
      ▲
[DaemonSet: svc-* pods on each node]

How it works: For each LoadBalancer Service, Klipper creates a DaemonSet with svc- prefixed pods that bind to the host port. The node's own external IP is reported as the EXTERNAL-IP. There is no IP announcement to the network — it simply binds ports.

K3s-specific note: Klipper is enabled by default. To run MetalLB or any other LB controller, you must disable it:

# During K3s install
curl -sfL https://get.k3s.io | sh -s - --disable servicelb

# Or in K3s config file
disable:
  - servicelb

Feature	Rating
Zero config	✅ Built-in
True IP announcement	❌ No
BGP support	❌ No
Multi-node HA	⚠️ Failover only
Production-readiness	⚠️ Dev/small clusters
Resource usage	✅ Minimal

Best for: Local dev, single-node K3s, homelab, quick demos.

2. 🟢 NGINX Ingress Controller

What it is: The most widely deployed Kubernetes Ingress controller, based on the battle-tested NGINX reverse proxy. Two major variants exist: the community ingress-nginx and the commercial NGINX Inc. version (nginx-ingress).

Architecture:

Internet
   │
   ▼
[NGINX Pod]
   │  Reads Ingress rules + Annotations
   ├──► /app-a  ──► Service A ──► Pods
   ├──► /app-b  ──► Service B ──► Pods
   └──► /api    ──► Service C ──► Pods
        │
   [ConfigMap / Annotations drive nginx.conf]

Key features:

Annotation-driven configuration (granular control via nginx.ingress.kubernetes.io/*)
SSL termination, wildcard certs, HSTS
Rate limiting, IP allowlisting, custom error pages
WebSocket support, gRPC proxying
Prometheus metrics out of the box
ModSecurity WAF support (community build)

K3s installation:

# First, disable K3s's default Traefik if you want NGINX instead
curl -sfL https://get.k3s.io | sh -s - --disable traefik

# Install NGINX Ingress via Helm
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace

Sample Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/rate-limit: "100"
spec:
  ingressClassName: nginx
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-app-svc
            port:
              number: 80

Performance: NGINX processes ~30,000–40,000 RPS per instance in typical Kubernetes ingress scenarios. Config reloads happen on Ingress updates (brief traffic disruption is possible on busy clusters).

Feature	Rating
Community & docs	✅ Massive
Annotation flexibility	✅ Excellent
Auto TLS (Let's Encrypt)	⚠️ Needs cert-manager
Dynamic config (no reload)	❌ Requires reload
Performance	✅ Very good
K3s compatibility	✅ Excellent
Learning curve	✅ Low

Best for: Teams migrating from traditional NGINX setups, production HTTP/HTTPS workloads, teams needing extensive annotation-based customization.

3. 🐹 Traefik (K3s Default)

What it is: A cloud-native reverse proxy and ingress controller written in Go. K3s ships Traefik v2 by default (upgraded to v3 in recent K3s releases). It auto-discovers services via Kubernetes CRDs and annotations.

Architecture:

Internet
   │
   ▼
[Traefik Proxy]
   │  Watches: IngressRoutes, Ingress, Services
   │  Providers: Kubernetes CRD, Kubernetes Ingress
   │
   ├─[Routers]──[Middlewares]──[Services]──► Pods
   │     │            │
   │  Host/Path    RateLimit
   │  rules        Auth
   │               Retry
   │
   └─[Dashboard: :8080]  [Metrics: Prometheus]

Key features:

Zero-config service discovery — annotate a Service and Traefik picks it up instantly, no config file reloads
Automatic Let's Encrypt TLS with ACME challenge support
Middleware system: auth, rate limiting, headers, circuit breakers, retry
Native IngressRoute CRDs for full power
Built-in dashboard and Prometheus metrics
TCP/UDP routing support (not just HTTP)

K3s-specific note: Traefik is bundled and managed by K3s. To customize it, use a HelmChartConfig:

# /var/lib/rancher/k3s/server/manifests/traefik-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    dashboard:
      enabled: true
    additionalArguments:
      - "--entrypoints.websecure.http.tls"
    ports:
      web:
        redirectTo: websecure

Sample IngressRoute:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: my-app
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`myapp.example.com`)
    kind: Rule
    services:
    - name: my-app-svc
      port: 80
    middlewares:
    - name: rate-limit
  tls:
    certResolver: letsencrypt

Performance: Traefik handles ~19,000 RPS with very stable resource consumption and zero-reload dynamic config — a key advantage over NGINX for fast-moving microservices.

Feature	Rating
K3s integration	✅ Native, bundled
Auto TLS (Let's Encrypt)	✅ Built-in ACME
Dynamic config (no reload)	✅ Real-time
Dashboard	✅ Built-in
TCP/UDP routing	✅ Yes
Performance vs NGINX	⚠️ Slightly lower RPS
Enterprise features	⚠️ Enterprise version needed

Best for: K3s default stack, teams wanting zero-touch TLS, GitOps-friendly pipelines, dev-friendly environments.

4. 🔷 MetalLB

What it is: A bare-metal L4 load balancer for Kubernetes. It gives LoadBalancer type Services an actual external IP from a pool you define, using either Layer 2 (ARP) or BGP protocols.

Architecture (Layer 2 mode):

External Network
      │
      │  ARP: "Who has 192.168.1.100?" → Leader Node replies
      ▼
[Leader Node] ──► kube-proxy ──► Service Pods (all nodes)
      │
[MetalLB Speaker DaemonSet] on every node
[MetalLB Controller] handles IP assignment

Architecture (BGP mode):

[Router/Switch]
      │  BGP peering
      ▼
[MetalLB Speaker] on each K3s node
      │  Announces /32 routes per service IP
      ▼
[Direct packet routing to node]

K3s installation:

# Step 1: Disable Klipper
curl -sfL https://get.k3s.io | sh -s - --disable servicelb

# Step 2: Install MetalLB
helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb -n metallb-system --create-namespace

# Step 3: Configure IP pool
kubectl apply -f - <<EOF
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: k3s-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.200-192.168.1.220
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: k3s-l2
  namespace: metallb-system
EOF

Important caveat: In L2 mode, MetalLB doesn't truly load-balance at L4 — it elects a leader node that handles ARP for a given IP, and kube-proxy does the actual pod distribution. It's more of a failover mechanism than a true LB. BGP mode provides real per-node distribution but requires BGP-capable routers.

Feature	Rating
Bare-metal IP assignment	✅ Core purpose
BGP mode	✅ Yes
Layer 2 mode	✅ Yes (ARP/NDP)
True L4 load balancing	⚠️ BGP only
K3s compatibility	✅ Excellent (disable Klipper first)
Resource usage	✅ Very lightweight
Requires routers	⚠️ BGP mode does

Best for: Bare-metal K3s clusters that need proper external IPs, homelab with a VLAN IP pool, edge deployments without cloud LB.

5. ⚡ HAProxy Ingress Controller

What it is: The Kubernetes ingress controller backed by HAProxy — historically the gold standard for raw TCP/HTTP load balancing performance. HAProxy Technologies' own benchmarks show their ingress controller handling 42,000 RPS with the lowest CPU among all competitors.

Architecture:

Internet
   │
   ▼
[HAProxy Pod]
   │  Config generated from Ingress/CRDs by controller
   │
   ├─[Frontend: bind *:80]
   │       │
   │  [ACL rules: path_beg, hdr_dom]
   │       │
   └─[Backend pools] ──► Pod endpoints (health-checked)
         │
   [Stats: :1936]  [Prometheus metrics]

Key features:

Best-in-class raw throughput and lowest latency at scale
Native support for HTTP/3, QUIC, gRPC
Fine-grained connection control (timeouts, retries, stick tables)
Advanced Layer 7 routing: headers, cookies, ACLs
TCP mode for non-HTTP workloads
Gateway API support (HAProxy Ingress Controller v3.1+)

K3s installation:

helm repo add haproxytech https://haproxytech.github.io/helm-charts
helm install haproxy-ingress haproxytech/kubernetes-ingress \
  --namespace haproxy-controller --create-namespace \
  --set controller.service.type=LoadBalancer

Performance edge: In head-to-head benchmarks against NGINX, Traefik, and Envoy:

HAProxy: 42,000 RPS, 50% CPU
NGINX: ~35,000 RPS, ~65% CPU
Traefik: ~19,000 RPS, ~45% CPU (more consistent)
Envoy: ~38,000 RPS, 73% CPU

Feature	Rating
Raw throughput	✅ Best-in-class
HTTP/3 & gRPC	✅ Yes
Advanced ACLs	✅ Very powerful
Auto TLS	⚠️ Needs cert-manager
Dynamic config	✅ v2.4+ hitless reload
K3s compatibility	✅ Good
Complexity	⚠️ Steeper learning curve

Best for: High-throughput production clusters, financial services, teams needing ultra-low p99 latency, TCP-heavy workloads.

6. 🌊 Envoy Proxy

What it is: Originally built at Lyft, Envoy is a high-performance C++ proxy that has become the de facto data plane of the cloud-native ecosystem. It powers Istio, Consul Connect, AWS App Mesh, and is the backbone of the Kubernetes Gateway API ecosystem.

Architecture:

[xDS Control Plane] (e.g., Istio's istiod)
       │  gRPC streaming: LDS, RDS, CDS, EDS
       ▼
[Envoy Proxy Instance]
   │
   ├─ Listeners (ports/protocols)
   │       │
   │  Filter Chains (HTTP, TCP, gRPC filters)
   │       │
   └─ Clusters (upstream endpoints)
         │
      [Circuit Breaker] [Retry] [Outlier Detection]

Key features:

Dynamic configuration via xDS API (zero-downtime updates)
Built-in circuit breaking, retries, outlier detection
Excellent observability: detailed stats, tracing (Zipkin/Jaeger/OTLP), access logs
gRPC-first with HTTP/1.1 and HTTP/2 support
Mutual TLS (mTLS) between services
WebAssembly (Wasm) plugin extensibility
Rate limiting via external services (Ratelimit service)

Standalone on K3s (without Istio):

# Envoy Gateway — standalone Gateway API implementation
helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.2.0 -n envoy-gateway-system --create-namespace

Performance: Envoy delivers ~38,000 RPS with excellent handling of dynamic service churn (critical for microservices that scale up/down frequently). Its sub-10ms latency during pod scaling events makes it ideal for Netflix/Uber-style workloads.

Feature	Rating
Dynamic config (xDS)	✅ Best-in-class
Observability	✅ Exceptional
gRPC support	✅ Native
Circuit breaking	✅ Built-in
Wasm extensibility	✅ Yes
Standalone complexity	⚠️ High (needs control plane)
K3s standalone use	⚠️ Via Envoy Gateway

Best for: Microservices architectures with dynamic service discovery, service mesh data plane, teams that need xDS-compatible control plane integration.

7. 🕸️ Istio (Service Mesh)

What it is: The most feature-complete service mesh for Kubernetes. Istio injects Envoy sidecars into every pod and manages the entire service-to-service communication layer via a centralized control plane (istiod).

Architecture:

[istiod - Control Plane]
   ├── Pilot (traffic management)
   ├── Citadel (certificate authority)
   └── Galley (config validation)
         │  xDS API
         ▼
[Pod A]                    [Pod B]
  App Container              App Container
  Envoy Sidecar ◄──mTLS──► Envoy Sidecar
  (intercepts all traffic)   (intercepts all traffic)

Istio Ambient Mode (2024/2026): The new sidecar-free mode using per-node "ztunnel" proxies + optional Waypoint proxies eliminates the double-hop latency, bringing performance near bare-metal levels.

Key features:

Fine-grained traffic management: canary, A/B, weighted routing, fault injection
Automatic mTLS between all services
Authorization policies at L7 (RBAC per HTTP path/method)
Distributed tracing, Kiali topology visualization
Multi-cluster and VM support
Gateway API support

K3s resource requirements (important!):

istiod: ~500MB RAM
Per-pod Envoy sidecar: ~50MB RAM each
At 500 services: 25–50GB extra RAM vs. Linkerd — plan accordingly

# Install Istio on K3s
curl -L https://istio.io/downloadIstio | sh -
istioctl install --set profile=minimal -y
kubectl label namespace default istio-injection=enabled

Feature	Rating
Traffic management	✅ Most advanced
mTLS	✅ Automatic
Observability	✅ Full stack (Kiali, Jaeger)
Authorization policies	✅ L7 RBAC
Resource usage	❌ Heavy (per-pod sidecar)
Complexity	❌ High
K3s (small cluster)	⚠️ Feasible, watch RAM

Best for: Enterprise Kubernetes, SOC 2/PCI-DSS compliance requirements, teams needing canary deployments and fault injection, hybrid VM+K8s environments.

8. 🔗 Linkerd (Service Mesh)

What it is: The original service mesh (coined the term in 2016). Linkerd uses a Rust-based "microproxy" instead of Envoy — dramatically lighter weight, making it the fastest and most resource-efficient service mesh available.

Architecture:

[Linkerd Control Plane]
  ├── destination (service discovery)
  ├── identity (certificate authority)
  └── proxy-injector (sidecar injection)
         │
[Pod A]                    [Pod B]
  App Container              App Container
  linkerd2-proxy ◄──mTLS──► linkerd2-proxy
  (Rust, ~10MB RAM each)     (tiny overhead!)

Performance benchmarks (vs other meshes):

Linkerd: ~5–10% slower than baseline (no mesh) — best among all meshes
Istio: ~25–35% slower than baseline
Cilium Mesh: ~20–30% slower than baseline

Key features:

Automatic mTLS (on by default, zero config)
Golden signals dashboard (latency, traffic, errors, saturation)
Per-route metrics
Traffic splitting (canary, A/B)
Multi-cluster support
FIPS-compliant builds available
Graduated CNCF project (most mature after Istio)

K3s installation:

# Install Linkerd CLI
curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/install | sh

# Pre-flight check
linkerd check --pre

# Install on K3s
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
linkerd check

# Inject into a namespace
kubectl annotate namespace default linkerd.io/inject=enabled

Feature	Rating
Resource efficiency	✅ Best among meshes
Performance overhead	✅ Minimal (5–10%)
mTLS	✅ Auto, zero-config
Simplicity	✅ Easiest mesh
Dashboard	✅ Built-in
Advanced traffic routing	⚠️ Less than Istio
K3s compatibility	✅ Excellent

Best for: Teams wanting mesh capabilities without Istio's complexity, K3s clusters with limited RAM, security-first teams, anyone who wants to "just turn it on and have it work."

9. 🧬 Cilium (eBPF-based CNI + Service Mesh)

What it is: Cilium is fundamentally different from all others — it operates at the Linux kernel level using eBPF (extended Berkeley Packet Filter), replacing traditional iptables networking entirely. It serves as both a CNI (network plugin) and optionally a service mesh.

Architecture:

[Cilium Operator] + [Cilium Agent DaemonSet]
         │  Programs eBPF maps
         ▼
[Linux Kernel - eBPF programs]
   ├── XDP (eXpress Data Path): packet filtering at NIC level
   ├── TC (Traffic Control): L3/L4 policy enforcement
   └── Socket: L7 visibility (HTTP, gRPC, Kafka, DNS)
         │
[Hubble Observability Layer]
   ├── hubble-relay
   └── hubble-ui (real-time network flow visualization)

Key features:

eBPF-powered networking: bypasses kernel overhead, hardware-speed L4
No iptables — replaces kube-proxy entirely
Deep observability via Hubble (DNS, HTTP, gRPC, Kafka at kernel level)
Network policies at L3/L4/L7 in a single CRD
WireGuard/IPsec transparent encryption
Service mesh in per-node Envoy model (not sidecar-per-pod)
Excellent for multi-cluster with Cluster Mesh

K3s installation:

# Disable K3s's default flannel (Cilium replaces it)
curl -sfL https://get.k3s.io | sh -s - \
  --flannel-backend=none \
  --disable-network-policy \
  --disable servicelb

# Install Cilium
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium \
  --namespace kube-system \
  --set operator.replicas=1 \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=<K3S_SERVER_IP> \
  --set k8sServicePort=6443

# Enable Hubble
cilium hubble enable --ui

L4 performance: Cilium's eBPF datapath is unrivaled for L4 (TCP/UDP) — limited only by hardware NIC speed. For L7 (HTTP), it offloads to per-node Envoy, which introduces some trade-offs vs. per-pod sidecar isolation.

Feature	Rating
L4 throughput	✅ Best (eBPF)
Network observability	✅ Exceptional (Hubble)
No iptables	✅ kube-proxy replacement
Network policies	✅ L3/L4/L7 unified
Service mesh	⚠️ Per-node (not per-pod)
Complexity	⚠️ eBPF expertise needed
K3s integration	✅ Good (replaces flannel)

Best for: High-performance bare-metal clusters, security-intensive environments, teams already investing in eBPF, multi-cluster deployments with Cluster Mesh.

📊 The Big Comparison Table

Tool	Type	OSI Layer	K3s Default	Auto TLS	Performance	Resource Usage	Complexity
Klipper/ServiceLB	L4 LB	L4	✅ Yes	❌	Low	Minimal	Minimal
NGINX	Ingress	L7	❌ (opt-out Traefik)	⚠️ (cert-manager)	Very High	Low	Low
Traefik	Ingress	L7	✅ Yes (bundled)	✅ Built-in	High	Low	Low
MetalLB	L4 LB	L4	❌	❌	Medium	Minimal	Low
HAProxy	Ingress	L4+L7	❌	⚠️ (cert-manager)	Highest	Low	Medium
Envoy	Proxy/Mesh DP	L4+L7	❌	✅ (with CP)	Very High	Medium	High
Istio	Service Mesh	L4+L7	❌	✅ Auto mTLS	Medium (overhead)	Very High	Very High
Linkerd	Service Mesh	L4+L7	❌	✅ Auto mTLS	High (least overhead)	Low	Low
Cilium	CNI+Mesh	L3+L4+L7	❌	✅ (WireGuard)	Highest L4	Medium	High

🏗️ Architecture Patterns for K3s

Pattern 1: Minimal (Single Node / Homelab)

[K3s: Traefik + Klipper built-in]
   │
   └── Just works. Zero extra config needed.

Use when: Local dev, single-node homelab, learning Kubernetes.

Pattern 2: Bare-Metal Production (Most Common)

[MetalLB] ──► External IP ──► [Traefik] ──► [Your Services]

Use when: Multiple K3s nodes, need proper external IPs, keep Traefik for simplicity.

Pattern 3: High-Performance Production

[MetalLB] ──► External IP ──► [HAProxy Ingress] ──► [Services]

Use when: High RPS requirements, latency-sensitive APIs, financial/gaming workloads.

Pattern 4: Secure Microservices (Security-First)

[MetalLB] ──► [NGINX/Traefik] ──► [Linkerd Mesh] ──► [Services]
                                      (mTLS, observability)

Use when: Multi-service architecture, compliance requirements, need service-to-service encryption.

Pattern 5: Maximum Performance + Security (Advanced)

[Cilium CNI + kube-proxy replacement]
   └──► [Cilium Ingress / Envoy Gateway] ──► [Services]
        + Hubble for observability

Use when: eBPF expertise available, need kernel-level performance, security-intensive platform.

🏎️ Performance Benchmarks at a Glance

Based on published benchmarks and production data (2024–2026):

Requests per Second (RPS) at typical K8s ingress workload:

HAProxy    ████████████████████████████  42,000 RPS  (50% CPU)
Envoy      ███████████████████████████   38,000 RPS  (73% CPU)
NGINX      ██████████████████████████    35,000 RPS  (65% CPU)
Traefik    █████████████                 19,000 RPS  (45% CPU)

Service Mesh Overhead (vs no mesh):
Linkerd    ██  5–10% slower   ← Best
Cilium     ████  20–30% slower
Istio      █████  25–35% slower

L4 Raw Throughput:
Cilium (eBPF)  ████████████████████  Hardware-limited ← Best
MetalLB (BGP)  ██████████████████    Near line-rate

🎯 Decision Framework: Which One for Your K3s Cluster?

START HERE
    │
    ▼
Are you running a single node / homelab?
  YES ──► Use Klipper + Traefik (K3s defaults). You're done.
  NO
    │
    ▼
Do you need external IPs on bare metal?
  YES ──► Add MetalLB (disable Klipper first)
  NO (cloud) ──► Your cloud CCM handles this
    │
    ▼
Replace default Traefik ingress?
  Need max performance ──► HAProxy Ingress
  Need NGINX ecosystem ──► NGINX Ingress
  Happy with defaults   ──► Keep Traefik
    │
    ▼
Do you have multiple microservices needing service-to-service security?
  YES, want simplicity ──► Add Linkerd
  YES, need full features ──► Add Istio (check your RAM budget!)
  YES, eBPF expertise ──► Use Cilium as CNI + mesh
  NO ──► Skip the mesh for now

🔧 K3s-Specific Tips & Gotchas

Traefik version: K3s bundles Traefik. Pin the version in your HelmChartConfig if stability matters.
MetalLB + Traefik: A very common combo. MetalLB gives Traefik a real external IP. After MetalLB assigns an IP, Traefik's LoadBalancer service gets EXTERNAL-IP populated and starts serving traffic.
Cilium on K3s: You must disable flannel (--flannel-backend=none) and network policy (--disable-network-policy). Cilium replaces both. If you also want to replace kube-proxy, add --disable-kube-proxy.
Linkerd on K3s: Works out of the box. K3s's bundled components (Traefik, CoreDNS) can be meshed too — annotate the kube-system namespace carefully.
Resource planning: A 3-node K3s cluster with Linkerd can run comfortably on 3× Raspberry Pi 4 (4GB). Istio needs significantly more — budget at least 8GB per node.
Gateway API: The Kubernetes Gateway API is replacing Ingress. Traefik v3, HAProxy v3.1+, Envoy Gateway, and Cilium all support it. Consider Gateway API for new deployments.

🏁 Final Recommendations

Your Situation	Recommended Stack
Homelab / learning	K3s defaults (Traefik + Klipper)
Bare-metal small team	MetalLB + Traefik
Bare-metal high traffic	MetalLB + HAProxy
NGINX ecosystem familiarity	MetalLB + NGINX Ingress
Need service mesh (simple)	MetalLB + Traefik + Linkerd
Need service mesh (full features)	MetalLB + Traefik + Istio (Ambient mode)
Max performance + security	Cilium CNI + Envoy Gateway
Edge/IoT K3s	Klipper + Traefik (minimal resources)

📚 Further Reading

Have questions about your specific K3s setup? Drop them in the comments. Running an unusual configuration (Raspberry Pi cluster, edge IoT, air-gapped)? I'd love to hear about it.

#kubernetes #k3s #devops #cloudnative #loadbalancing #traefik #nginx #metallb #linkerd #cilium

Kubernetes CNI Complete Guide: Flannel vs Cilium vs Calico + Cloud Provider CNIs

Pendela BhargavaSai — Tue, 12 May 2026 03:30:00 +0000

K3s v1.29+ | Flannel v0.24+ | Cilium v1.15+ | Calico v3.27+ | AWS VPC CNI v1.18+ | Azure CNI v1.5+ | GKE Dataplane V2 (Cilium-based)

A definitive comparison of every major Kubernetes CNI — open-source plugins (Flannel, Calico, Cilium, Weave, Antrea, Multus) and cloud-managed defaults (AWS VPC CNI on EKS, Azure CNI on AKS, and GKE's Dataplane V2 on GKE) — across architecture, performance, network policy, observability, encryption, and when to choose each.

CNI	Identity	Core Approach	Default On
🟢 Flannel	Simple Overlay	VXLAN tunnel, zero policy	K3s
🟠 Calico	Policy Powerhouse	BGP routing, iptables/eBPF	Self-managed
🔵 Cilium	eBPF Native	Kernel eBPF, replaces kube-proxy	GKE (Dataplane V2)
🟡 Weave Net	Mesh Overlay	Gossip-based mesh routing	Self-managed
🟣 Antrea	VMware-backed	OVS dataplane, Antrea policies	Self-managed
🔶 AWS VPC CNI	Cloud-native	Native VPC IP assignment	EKS
🔷 Azure CNI	Cloud-native	Azure VNET IP assignment	AKS
♦️ GKE CNI / Dataplane V2	Cloud-native + eBPF	Cilium-based eBPF on GKE	GKE

What Is a CNI?
Open Source CNIs
- 2.1 Flannel — Simple Overlay
- 2.2 Cilium — eBPF Native
- 2.3 Calico — BGP + Flexible Dataplane
- 2.4 Weave Net — Mesh Overlay
- 2.5 Antrea — OVS-based CNI
- 2.6 Multus — Meta CNI
Cloud Provider CNIs
- 3.1 AWS VPC CNI — EKS Default
- 3.2 Azure CNI — AKS Default
- 3.3 GKE Dataplane V2 — GKE Default
Data Plane Comparison
Network Policy
Observability
Performance Benchmarks
Encryption
Multi-Cluster
Resource Usage
Full Feature Comparison
When to Choose Each
K3s-Specific Setup
Migration Guide on K3s
Conclusion

1. What Is a CNI and Why Does It Matter?

The Container Network Interface (CNI) is the plugin layer every Kubernetes cluster depends on for:

Assigning IP addresses to pods from a defined CIDR range
Creating virtual Ethernet (veth) pairs between pod namespaces and the host
Programming cross-node routing so pods on Node A can reach pods on Node B
Optionally enforcing NetworkPolicy resources to control traffic flow

Cloud providers like AWS, Azure, and GCP have built proprietary CNI plugins that deeply integrate with their underlying VPC/VNET networking primitives — providing native IP assignment, cloud-aware routing, and tight integration with cloud IAM, load balancers, and security groups.

💡 K3s Key Flag
To replace the default CNI on K3s, install with --flannel-backend=none --disable-network-policy. This leaves the CNI slot open for Calico or Cilium to fill.

2. Open Source CNIs

2.1 Flannel Simple Overlay

Flannel's design philosophy: do one thing well. A user-space daemon (flanneld) manages subnet allocation, while the kernel's own VXLAN and bridge code handles all actual forwarding. No policy, no observability — just connectivity.

Pod A (eth0: 10.244.0.2)          Pod B (eth0: 10.244.0.5)
        │                                  │
        │ veth pair                        │ veth pair
        ▼                                  ▼
           cni0 Linux bridge (kernel)
                    │
      iptables PREROUTING / FORWARD / POSTROUTING
                    │
         VXLAN encapsulation — UDP 8472
                    │
     flanneld (user-space) ← etcd / K8s API
                    │
          Physical NIC → Node B

Available backends:

Backend	Transport	Use Case
`vxlan`	UDP encap (default)	Works across any network, even routers
`host-gw`	Direct routing	Fastest, requires L2 adjacency between nodes
`wireguard-native`	Encrypted WireGuard tunnel	When you need encryption
`udp`	Legacy user-space	Fallback only — very slow

Network Policy: Flannel enforces zero NetworkPolicy. Resources are silently ignored. You must pair it with Calico (Canal) to get policy — adding a second DaemonSet, version compatibility risk, and split ownership between two projects.

Flannel Encryption: Flannel encrypts cross-node traffic only — pod-to-pod on the same node travels through the cni0 bridge unencrypted. No auto key rotation; restart flanneld to rotate keys.

{
  "Network": "10.244.0.0/16",
  "Backend": {
    "Type": "wireguard"
  }
}

Best for: Dev/CI clusters, Raspberry Pi, edge nodes, K3s defaults.

2.2 Cilium — eBPF Native

Cilium compiles and injects eBPF programs into the Linux kernel at TC/XDP hook points. There is no bridge, no iptables — packets are forwarded via bpf_redirect() at line rate, and policy is enforced via O(1) BPF map lookups.

Pod A (eth0)                         Pod B (eth0)
       │                                  │
       │ veth pair                        │
       ▼                                  ▼
TC eBPF hook ──── bpf_redirect() ──── TC eBPF hook
                  │
BPF maps: identity · policy · NAT · LB
                  │
cilium-agent — compiles eBPF, watches K8s API
                  │
  Physical NIC — GENEVE / native routing

Datapath modes:

Mode	Encapsulation	Requirement
`tunnel: geneve`	GENEVE (default)	Any network topology
`native-routing`	None	L2 adjacency or BGP underlay
`wireguard`	WireGuard transparent	Kernel ≥ 5.6
`ipsec`	IPsec	FIPS-regulated environments

Network Policy: 4.3 Cilium — L3 Through L7, No Sidecar

Cilium enforces standard NetworkPolicy and extends it with CiliumNetworkPolicy (CNP) for Layer 7 rules — no sidecar required:

# CiliumNetworkPolicy — L7 HTTP rule
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-get-only
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: "/api/v1/.*"

🔭 Cilium + Hubble

✅ Per-flow visibility on every packet
✅ Live service dependency map (Hubble UI)
✅ L7 HTTP / DNS / Kafka / gRPC flows
✅ Drop reason per endpoint
✅ Rich Prometheus metrics

# Enable Hubble and UI
cilium hubble enable --ui

# Watch live flows in a namespace
hubble observe --namespace production --follow

# Show only policy drops with reason
hubble observe --verdict DROPPED --follow

# Sample output:
# 12:34:01: default/frontend → default/backend  FORWARDED  TCP:SYN
# 12:34:02: default/attacker → default/backend  DROPPED    Policy denied

Cilium Encryption: Cilium WireGuard + IPsec

# WireGuard with strict mode (drops unencrypted packets)
cilium install \
  --encryption wireguard \
  --encryption-strict-mode true

# IPsec for FIPS-regulated environments
cilium install --encryption ipsec

Best for: Large-scale production, L7 policy, observability (Hubble), zero-trust, multi-cluster.

2.3 Calico — BGP + Flexible Dataplane

Calico uses BGP (Border Gateway Protocol) to distribute pod routes across nodes — no encapsulation by default. Each node acts as a BGP peer, advertising its pod CIDR to other nodes and upstream routers. Calico's data plane is pluggable: iptables, eBPF, or even Windows HNS.

Pod A (eth0: 192.168.0.2)          Pod B (eth0: 192.168.1.2)
        │                                  │
        │ veth pair                        │ veth pair
        ▼                                  ▼
      Host routing table (no bridge needed)
                    │
      iptables / eBPF policy enforcement
                    │
     Felix (per-node agent) ← Typha (fan-out)
                    │
     BIRD (BGP daemon) — peers with other nodes
                    │
    Physical NIC — direct IP routing (no encap)

Key Calico components:

Component	Role
Felix	Per-node agent; programs iptables/eBPF rules and routes
BIRD	Open-source BGP daemon; advertises pod subnets to peers
Typha	Fan-out proxy for the K8s datastore; recommended at 50+ nodes
calico-kube-controllers	Garbage-collects stale Calico resources

Network Policy: 4.2 Calico — L3/L4 Policy Leader

Calico is widely regarded as the gold standard for L3/L4 NetworkPolicy. It supports standard NetworkPolicy resources plus its own GlobalNetworkPolicy and NetworkSet CRDs:

# Calico GlobalNetworkPolicy — cluster-wide deny-all
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: default-deny-all
spec:
  selector: all()
  types:
  - Ingress
  - Egress

# Calico NetworkSet — group external CIDRs
apiVersion: projectcalico.org/v3
kind: NetworkSet
metadata:
  name: trusted-external
spec:
  nets:
  - 203.0.113.0/24
  - 198.51.100.0/24

⚠️ Calico does not support L7 HTTP/gRPC policy natively in OSS. For that you need its optional Envoy-based Application Layer Policy (ALP), which adds a sidecar and complexity.

Calico Encryption: Calico supports WireGuard for node-to-node encryption, enabled with a single patch:

kubectl patch felixconfiguration default \
  --type merge \
  --patch '{"spec":{"wireguardEnabled":true}}'

Starting in Calico v3.26, same-node pod traffic encryption is also supported via host-to-pod WireGuard options.

Best for: BGP-integrated DCs, Windows node support, bare-metal L3, robust L3/L4 policy.

2.4 Weave Net — Mesh Overlay

Weave Net uses a gossip protocol to build a full mesh topology between all cluster nodes without any central store. It wraps packets in a sleeve (VXLAN-like) tunnel and can optionally encrypt all traffic with NaCl. Weave is simpler to operate than Calico/Cilium but is no longer under active development (archived by Weaveworks in 2023).

Pod A (eth0)
       │
    weave bridge
       │
  weave daemon (gossip mesh peer discovery)
       │
  Sleeve / Fast Datapath (VXLAN kernel bypass)
       │
    Node B weave daemon
       │
    Pod B (eth0)

Key characteristics:

Feature	Detail
Discovery	Gossip — no external etcd needed
Datapath	Sleeve (user-space) or Fast Datapath (kernel VXLAN)
Encryption	NaCl (enabled per-pod connection)
NetworkPolicy	✅ Standard K8s policy supported
Status	⚠️ Archived/maintenance mode (use Cilium or Calico for new clusters)

⚠️ Important: Weaveworks ceased active development in 2023. Weave Net is community-maintained but no longer receives feature updates. It is not recommended for new clusters — migrate to Cilium or Calico.

Best for: Legacy clusters already running Weave with migration on the roadmap.

2.5 Antrea — OVS-based CNI

Antrea is a CNI backed by VMware (now Broadcom) that uses Open vSwitch (OVS) as its dataplane. It supports both Linux and Windows nodes and provides its own AntreaNetworkPolicy and ClusterNetworkPolicy CRDs with tiered policy enforcement. Antrea integrates well with NSX-T for enterprise SD-WAN environments.

Pod A (eth0)
       │
   OVS (Open vSwitch) bridge
       │
   antrea-agent (per-node DaemonSet)
       │
   antrea-controller (centralized)
       │
   Encap: Geneve / VXLAN / GRE (configurable)
       │
   Node B OVS bridge → Pod B

Key features:

Feature	Antrea
Dataplane	Open vSwitch (OVS)
Windows support	✅ Full (OVS on Windows)
NetworkPolicy	✅ K8s standard + AntreaNetworkPolicy CRDs
Tiered policy	✅ (Emergency / Security / Application tiers)
Encryption	✅ IPsec / WireGuard
Observability	✅ Antrea Octant plugin, Prometheus metrics
NSX-T integration	✅ Enterprise add-on
eBPF support	✅ AntreaProxy (partial eBPF)

Best for: VMware/NSX-T environments, Windows-heavy clusters, tiered network policy.

2.6 Multus — Meta CNI

Multus is not a standalone CNI — it is a meta CNI that allows pods to attach multiple network interfaces simultaneously. A pod can have its primary network (managed by Flannel/Calico/Cilium) and secondary interfaces (SR-IOV, DPDK, Macvlan) for specialized workloads like telco NFV or HPC.

Pod with Multiple NICs:
  eth0 (primary) ← Flannel/Calico/Cilium (cluster network)
  net1 (secondary) ← SR-IOV (high-throughput direct NIC)
  net2 (secondary) ← Macvlan (storage network)

Multus reads NetworkAttachmentDefinition CRDs and delegates
to the correct CNI for each interface.

# NetworkAttachmentDefinition for secondary interface
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: sriov-net
spec:
  config: |
    {
      "type": "sriov",
      "name": "sriov-net",
      "ipam": { "type": "static" }
    }

Best for: Telco/NFV workloads, HPC, pods that need to straddle multiple network segments.

3. Cloud Provider CNIs

Cloud-managed Kubernetes services ship their own CNI plugins that are deeply integrated with the underlying cloud networking fabric. These provide first-class VPC routing, cloud IAM integration, and managed lifecycle — but are typically locked to their respective cloud.

3.1 AWS VPC CNI — EKS Default

Amazon EKS uses the Amazon VPC CNI plugin (aws-node DaemonSet) by default. Instead of an overlay, it assigns real VPC secondary IP addresses directly to pods from Elastic Network Interfaces (ENIs) attached to the worker node.

Worker Node (EC2 instance)
    │
    ├── Primary ENI (node IP: 10.0.1.10)
    │      └── eth0
    │
    ├── Secondary ENI (attached by vpc-cni)
    │      ├── 10.0.1.20 → Pod A (eth0 via veth)
    │      ├── 10.0.1.21 → Pod B (eth0 via veth)
    │      └── 10.0.1.22 → Pod C (eth0 via veth)
    │
    └── vpc-cni (aws-node DaemonSet)
           manages ENI lifecycle via EC2 API

How pod IPs work:

Each EC2 instance can attach multiple ENIs; each ENI holds multiple secondary IPs
vpc-cni pre-warms a pool of secondary IPs per node via EC2 API calls
Pods receive a real VPC IP — routable natively across the VPC, peered VPCs, VPNs, and Direct Connect — with no overlay

Pod density limits per node (examples):

Instance Type	Max ENIs	Max IPs (pod limit)
t3.medium	3	17
m5.large	3	29
m5.xlarge	4	58
m5.4xlarge	8	234
c5.18xlarge	15	750

⚠️ Important: Default pod density is capped by the ENI/IP limit per instance type. For IP-constrained environments, use VPC CNI with prefix delegation (ENABLE_PREFIX_DELEGATION=true) to assign /28 prefixes instead of individual IPs, dramatically increasing pod density.

Key features:

Feature	AWS VPC CNI
IP assignment	Native VPC secondary IPs from ENIs
Overlay	✗ None — native VPC routing
NetworkPolicy	✗ Not built-in — requires Calico or Cilium add-on
Security Groups	✅ Security Groups for Pods (SGP) — per-pod AWS SGs
IPv6	✅ Supported
Prefix delegation	✅ /28 prefix per ENI (more pods per node)
Windows nodes	✅ Supported
Custom networking	✅ Pods in different subnet than node
eBPF acceleration	✅ via Cilium add-on (EKS + Cilium mode)

Enabling Network Policy on EKS:
AWS VPC CNI itself does not enforce NetworkPolicy. You must add one of:

Calico (most common) — install as an add-on alongside vpc-cni
Cilium in chained mode — replaces policy enforcement, keeps VPC IP routing
Amazon VPC CNI Network Policy (AWS-native, GA as of 2024) — uses eBPF for policy enforcement

# Enable AWS-native network policy controller (EKS add-on)
aws eks create-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --configuration-values '{"nodeAgent":{"enablePolicyEventLogs":"true"}}'

When to choose AWS VPC CNI:

✅ Running EKS — it is the default and AWS-managed
✅ Need pods directly reachable from on-premises via Direct Connect / VPN
✅ Need per-pod AWS Security Groups (SGP feature)
✅ Compliance requires no overlay network
⚠️ Watch instance type ENI limits for large pod densities

3.2 Azure CNI — AKS Default

Azure Kubernetes Service (AKS) offers multiple CNI modes. The default for most production clusters is Azure CNI, which assigns pod IPs directly from the Azure Virtual Network (VNET) subnet — similar in concept to AWS VPC CNI but using Azure's networking primitives.

AKS CNI Modes:

Mode	Description	Default?
kubenet	Basic overlay; nodes get VNET IPs, pods get private overlay IPs (NAT)	Legacy default
Azure CNI	Pods get real VNET IPs from a pre-allocated subnet	Current recommended default
Azure CNI Overlay	Pods get overlay IPs (larger scale, fewer VNET IPs needed)	Recommended for large clusters
Azure CNI + Cilium	Azure CNI routing + Cilium eBPF dataplane + Hubble	Recommended for policy/observability
Bring Your Own CNI	Disable Azure CNI; install Calico, Flannel, etc.	Advanced

Azure CNI (traditional):

AKS Worker Node (Azure VM)
    │
    ├── Primary NIC (node IP: 10.240.0.4)
    │      └── VNET: 10.240.0.0/16
    │
    └── Pod IPs pre-allocated from subnet:
           ├── 10.240.0.10 → Pod A
           ├── 10.240.0.11 → Pod B
           └── 10.240.0.12 → Pod C

azure-vnet (CNI plugin) programs routes in Azure SDN

Azure CNI Overlay (recommended for scale):
Introduced to solve IP exhaustion. Pods get IPs from a private overlay CIDR (e.g., 10.244.0.0/16) while nodes get real VNET IPs. Azure SDN handles the translation — no overlay encap at the packet level from the VM's perspective.

# Create AKS cluster with Azure CNI Overlay + Cilium dataplane
az aks create \
  --resource-group myRG \
  --name myAKS \
  --network-plugin azure \
  --network-plugin-mode overlay \
  --network-dataplane cilium \
  --pod-cidr 192.168.0.0/16

Key features:

Feature	kubenet	Azure CNI	Azure CNI Overlay	Azure CNI + Cilium
Pod IPs	Overlay (NAT)	Real VNET IPs	Overlay (Azure SDN)	Overlay (Azure SDN)
IP exhaustion risk	Low	High	Low	Low
Direct pod routing	✗	✅	✅ (via Azure SDN)	✅
NetworkPolicy	Basic	Azure Network Policy / Calico	Azure NP / Calico	✅ Cilium (eBPF)
Windows nodes	✅	✅	✅	⚠️ Partial
Hubble observability	✗	✗	✗	✅
Max pods/node	110	250	250	250

Network Policy options on AKS:

Azure Network Policy Manager (NPM) — iptables-based, Azure-native, limited feature set
Calico — add-on, full L3/L4 policy, most commonly used
Cilium — available with Azure CNI Overlay mode, eBPF enforcement + Hubble

When to choose Azure CNI:

✅ Running AKS — Azure CNI Overlay is the modern recommended choice
✅ Need pods directly reachable from on-premises via ExpressRoute
✅ Want Hubble observability → use Azure CNI Overlay + Cilium dataplane
✅ Large clusters (100+ nodes) → use Overlay mode to avoid VNET IP exhaustion
⚠️ Traditional Azure CNI requires pre-allocating pod IPs per node — plan subnet size carefully

3.3 GKE Dataplane V2 — GKE Default

Google Kubernetes Engine (GKE) introduced Dataplane V2 in 2021, which is based on Cilium's eBPF engine. It is the default for new GKE clusters and brings production-grade eBPF networking, built-in NetworkPolicy enforcement, and a subset of Hubble observability — all managed by Google.

GKE networking modes:

Mode	Description	Default?
Legacy (iptables)	kube-proxy + iptables, no Dataplane V2	Older clusters
Dataplane V2	Cilium eBPF, managed by GKE, no full Cilium control plane	Default for new clusters
Dataplane V2 + Hubble	Same + network telemetry via Hubble	Optional add-on

Architecture:

GKE Node (GCE VM)
    │
    ├── Alias IP range (VPC-native pod CIDRs)
    │     Pods get real VPC IPs, routed via Google SDN
    │
    └── Dataplane V2 (Cilium eBPF engine)
           ├── TC eBPF hooks on veth interfaces
           ├── BPF maps for policy, NAT, LB
           ├── kube-proxy replaced by eBPF
           └── Hubble telemetry (if enabled)

GKE uses VPC-native networking (alias IP ranges) — pods get real VPC CIDRs routed natively through Google's Andromeda SDN. Dataplane V2 sits on top, adding eBPF policy enforcement and observability.

Enabling Dataplane V2 on GKE:

# Create GKE cluster with Dataplane V2 (default for new clusters)
gcloud container clusters create my-cluster \
  --enable-dataplane-v2 \
  --enable-ip-alias \
  --location us-central1

# Enable Hubble observability add-on
gcloud container clusters update my-cluster \
  --enable-dataplane-v2-flow-observability \
  --location us-central1

Key features:

Feature	GKE Dataplane V2
Dataplane	Cilium eBPF (managed subset)
kube-proxy replacement	✅ eBPF
NetworkPolicy	✅ eBPF-enforced (L3/L4)
FQDN policy	✅ (GKE 1.28+)
Hubble observability	✅ Optional add-on
L7 policy	⚠️ Not exposed (managed limitations)
Pod IPs	Real VPC IPs (alias ranges)
Windows nodes	✅
Multi-cluster	✅ via GKE Fleet / Anthos
Managed lifecycle	✅ Google manages upgrades

Dataplane V2 vs self-managed Cilium on GKE:

Aspect	GKE Dataplane V2	Self-managed Cilium on GKE
Management	Google-managed	You manage Helm values/upgrades
Feature exposure	Subset of Cilium	Full Cilium feature set
Hubble	Basic (add-on)	Full Hubble UI + Relay
Cluster Mesh	✗ (use GKE Fleet)	✅
L7 CNP	✗	✅
Support	GKE SLA	Community / Isovalent

💡 GKE Recommendation: For most workloads, Dataplane V2 is the right choice — Google manages it, it's eBPF-based, and it covers L3/L4 policy. If you need full CiliumNetworkPolicy L7 rules or Cluster Mesh, consider self-managed Cilium on GKE with --network-plugin=cni and disabling kube-proxy.

When to choose GKE Dataplane V2:

✅ Running GKE — it is the default and Google-managed
✅ Want eBPF performance without managing Cilium yourself
✅ NetworkPolicy enforcement at scale (eBPF O(1) lookups)
✅ Need basic Hubble network telemetry
⚠️ For full L7 policy or Cluster Mesh, self-manage Cilium on GKE instead

4. Data Plane Comparison

Service Scalability — All CNIs

Services	Flannel (iptables)	Calico (iptables)	Calico (eBPF)	Cilium (eBPF)	AWS VPC CNI	Azure CNI	GKE DPv2
100	~10 ms	~10 ms	< 1 ms	< 1 ms	~10 ms	~10 ms	< 1 ms
1,000	~80 ms	~80 ms	< 1 ms	< 1 ms	~80 ms	~80 ms	< 1 ms
10,000	~800 ms	~800 ms	< 1 ms	< 1 ms	~800 ms	~800 ms	< 1 ms
50,000	⚠️ drops	⚠️ drops	< 1 ms	< 1 ms	⚠️ drops	⚠️ drops	< 1 ms

5. Network Policy

Policy Feature Comparison

Policy Feature	Flannel	Calico	Cilium	Weave	Antrea	AWS VPC CNI	Azure CNI	GKE DPv2
Standard NetworkPolicy	✗	✅	✅	✅	✅	✅ (add-on)	✅	✅
Egress Policy	✗	✅	✅	✅	✅	✅	✅	✅
GlobalNetworkPolicy	✗	✅	✅ CCNP	✗	✅ ClusterNetworkPolicy	✗	✗	✗
FQDN / DNS policy	✗	✅	✅	✗	✅	✗	✗	✅ (1.28+)
L7 HTTP method/path	✗	⚠️ ALP	✅ no sidecar	✗	✗	✗	✗	✗
Kafka / gRPC policy	✗	✗	✅	✗	✗	✗	✗	✗
Tiered policy	✗	✗	✗	✗	✅	✗	✗	✗
Security Groups (cloud)	✗	✗	✗	✗	✗	✅ SGP	✅ NSG	✅ Firewall rules

6. Observability

Feature	Flannel	Calico	Cilium	Weave	Antrea	AWS VPC CNI	Azure CNI	GKE DPv2
L3/L4 flow logs	✗	✅	✅	✗	✅	✅ VPC Flow Logs	✅ NSG Flow Logs	✅
L7 HTTP flows	✗	✗ (OSS)	✅	✗	✗	✗	✗	✗
Live service map	✗	✗	✅ Hubble UI	✗	✅ Octant	✗	✗	✅ (add-on)
Drop reason	✗	✅	✅	✗	✅	⚠️	⚠️	✅
Prometheus metrics	Basic	✅	✅ Rich	✅ Basic	✅	✅ CloudWatch	✅ Azure Monitor	✅
Built-in UI	✗	✗ (OSS)	✅ Hubble UI	✗	✅ Octant	✅ CloudWatch	✅ Azure Monitor	✅ Cloud Console

7. Performance Benchmarks

TCP Throughput — iperf3, Pod-to-Pod Same Node

CNI	Mode	Throughput
Flannel	VXLAN	~8 Gbps
Flannel	host-gw	~9.5 Gbps
Calico	BGP direct (iptables)	~9.3 Gbps
Calico	BGP direct (eBPF)	~9.7 Gbps
Cilium	GENEVE tunnel	~8.5 Gbps
Cilium	native-routing	~9.8 Gbps
Cilium	XDP	line rate
AWS VPC CNI	Native VPC routing	~9.5 Gbps
Azure CNI	Native VNET routing	~9.4 Gbps
GKE Dataplane V2	Alias IP + eBPF	~9.7 Gbps

⚠️ Results are representative — hardware, kernel version, and NIC driver all affect real-world numbers.

p99 Latency — Same Node

CNI	Mode	p99 Latency
Flannel	VXLAN	~0.35 ms
Flannel	host-gw	~0.18 ms
Calico	BGP direct (eBPF)	~0.15 ms
Cilium	native-routing	~0.16 ms
AWS VPC CNI	Native	~0.17 ms
Azure CNI	Native	~0.18 ms
GKE Dataplane V2	eBPF	~0.15 ms

8. Encryption

Feature	Flannel WG	Calico WG	Cilium WG	Cilium IPsec	Antrea WG/IPsec	AWS CNI	Azure CNI	GKE DPv2
Cross-node encryption	✅	✅	✅	✅	✅	✅ (NLB/TLS)	✅ (Azure Firewall)	✅ (WireGuard, beta)
Same-node encryption	✗	✅ (v3.26+)	✅	✅	✅	✗	✗	✗
Strict drop mode	✗	✗	✅	✅	✗	N/A	N/A	✗
Auto key rotation	✗	✅	✅	✅	✅	Managed	Managed	Managed
FIPS compliance	✗	✗	✗	✅	✅ IPsec	✅ (AWS FIPS)	✅ (Azure FIPS)	✅ (Google FIPS)

9. Multi-Cluster

Feature	Flannel	Calico	Cilium	Antrea	AWS EKS	Azure AKS	GKE
Native multi-cluster	✗	✅ BGP	✅ Cluster Mesh	✅ Antrea Multi-cluster	✅ EKS Connector	✅ AKS Fleet	✅ GKE Fleet
Unified service DNS	✗	✗	✅	✅	⚠️ (manual)	⚠️ (manual)	✅ (Anthos)
Cross-cluster NetworkPolicy	✗	✗ (OSS)	✅	✅	✗	✗	✅ (Anthos)
Cross-cluster observability	✗	✗	✅ Hubble	✅	✅ CloudWatch	✅ Azure Monitor	✅ Cloud Ops
Max clusters	—	Unlimited	255	Unlimited	Unlimited	Unlimited	Unlimited

10. Resource Usage

Resource	Flannel	Calico	Cilium	Weave	Antrea	AWS VPC CNI	Azure CNI	GKE DPv2
DaemonSet CPU (idle)	~5 mCPU	~20–60 mCPU	~30–80 mCPU	~10–30 mCPU	~20–50 mCPU	~10–25 mCPU	~10–30 mCPU	~30–80 mCPU
DaemonSet RAM (idle)	~30 MB	~60–150 MB	~100–300 MB	~50–100 MB	~50–100 MB	~30–80 MB	~40–80 MB	~100–300 MB
Startup time	~5s	~10–20s	~30–60s	~10s	~10–15s	~5–10s	~5–10s	Managed
Additional CRDs	0	~8	~15	0	~10	0–2	0	0
Minimum kernel	Any	Any / ≥5.3 (eBPF)	≥4.9	Any	Any	Any	Any	GKE-managed
Operator required	✗	✅ tigera	✅ cilium-operator	✗	✅ antrea-controller	AWS-managed	Azure-managed	GKE-managed

11. Full Feature Comparison

Dimension	Flannel	Calico	Cilium	Weave	Antrea	AWS VPC CNI	Azure CNI	GKE DPv2
Data plane	Bridge + iptables	BGP + iptables/eBPF	eBPF kernel-native	Mesh sleeve/VXLAN	OVS	VPC native	VNET native	eBPF (Cilium)
kube-proxy replacement	✗	✅ (eBPF)	✅	✗	✅ AntreaProxy	✗	✗	✅
Encapsulation	VXLAN	None/IPIP/VXLAN	GENEVE	Sleeve/VXLAN	Geneve/VXLAN	None	None	None
BGP routing	✗	✅ native	✅ optional	✗	✗	✗	✗	✗
L3/L4 NetworkPolicy	✗	✅	✅	✅	✅	✅ (add-on)	✅	✅
L7 HTTP/gRPC policy	✗	⚠️ ALP	✅ no sidecar	✗	✗	✗	✗	✗
FQDN-based policy	✗	✅	✅	✗	✅	✗	✗	✅ (1.28+)
GlobalNetworkPolicy	✗	✅	✅ CCNP	✗	✅ CNP	✗	✗	✗
Flow observability	✗	✅ flow logs	✅ Hubble	✗	✅ Octant	✅ VPC Flow	✅ NSG Flow	✅
L7 flow visibility	✗	✗ (OSS)	✅	✗	✗	✗	✗	✗
Cross-node encryption	✅ WG	✅ WG	✅ WG/IPsec	✅ NaCl	✅ WG/IPsec	Cloud-layer	Cloud-layer	✅ WG (beta)
Same-node encryption	✗	✅ (v3.26+)	✅	✗	✅	✗	✗	✗
FIPS encryption	✗	✗	✅ IPsec	✗	✅ IPsec	✅ (AWS)	✅ (Azure)	✅ (GCP)
Multi-cluster	✗	✅ BGP	✅ Cluster Mesh	✗	✅	EKS Fleet	AKS Fleet	GKE Fleet
Windows nodes	⚠️	✅ HNS	✗	✗	✅	✅	✅	✅
Cloud default	K3s	Manual	GKE	Manual	Manual	EKS	AKS	GKE
RAM per node (idle)	~30 MB	~60–150 MB	~100–300 MB	~50–100 MB	~50–100 MB	~30–80 MB	~40–80 MB	~100–300 MB
Operational complexity	Very low	Medium	Medium–High	Low	Medium	Low (managed)	Low (managed)	Low (managed)
Active development	✅	✅	✅	⚠️ Archived	✅	✅	✅	✅

12. When to Choose Each

🟢 Choose Flannel when…

✅ Dev, CI, or home lab cluster with no production traffic
✅ No NetworkPolicy requirement whatsoever
✅ RAM-constrained nodes (Raspberry Pi, 1 GB edge devices)
✅ You want the absolute lowest operational overhead
✅ Running a legacy kernel (RHEL 7 / CentOS 7)
✅ Already using a service mesh (Istio, Linkerd) for policy and observability

🟠 Choose Calico when…

✅ NetworkPolicy is required and Cilium feels like overkill
✅ You need BGP peering with upstream physical routers
✅ Windows nodes exist in your cluster
✅ No-encap direct routing is preferred for performance
✅ Your team already has Calico expertise
✅ Medium cluster size (10–200 nodes) with moderate policy complexity

🔵 Choose Cilium when…

✅ L7 HTTP/gRPC/Kafka policy without a service mesh sidecar
✅ Hubble observability and a live service map are needed
✅ 100+ services with high service churn (eBPF O(1) matters)
✅ End-to-end pod traffic encryption including same-node
✅ Multi-cluster federation with unified DNS and policy
✅ Building toward zero-trust networking inside the cluster

🟡 Choose Weave when…

⚠️ Generally not recommended for new clusters — Weaveworks is archived
✅ Only if migrating from an existing Weave deployment with no immediate migration path
✅ Simple overlay needed with built-in NaCl encryption (short term)

🟣 Choose Antrea when…

✅ VMware NSX-T / Tanzu environment requiring deep SD-WAN integration
✅ Tiered network policy enforcement (Emergency / Security / Application tiers)
✅ Windows and Linux mixed clusters in an enterprise VMware stack
✅ OVS dataplane is a hard requirement (telco, NFV)

🔶 Choose AWS VPC CNI (EKS) when…

✅ Running EKS — it is the default AWS-recommended CNI
✅ Pods must be natively routable across VPC, VPN, or Direct Connect
✅ Per-pod AWS Security Groups are required (SGP feature)
✅ Compliance mandates no overlay network
✅ Integrate with AWS services that need pod-level VPC routing

🔷 Choose Azure CNI (AKS) when…

✅ Running AKS — use Azure CNI Overlay mode for most production workloads
✅ Pods need to be reachable from on-prem via ExpressRoute
✅ Want eBPF performance + Hubble → choose Azure CNI Overlay + Cilium dataplane
✅ Large clusters → Azure CNI Overlay avoids VNET IP exhaustion
✅ Windows node support is required (all Azure CNI modes support it)

♦️ Choose GKE Dataplane V2 (GKE) when…

✅ Running GKE — it is the default for new clusters
✅ Want eBPF-based policy without managing Cilium yourself
✅ Need Hubble network telemetry (enable as add-on)
✅ FQDN-based NetworkPolicy (GKE 1.28+)
✅ Google-managed lifecycle and upgrades are preferred
⚠️ For L7 CNP or Cluster Mesh, self-manage Cilium on GKE instead

13. K3s-Specific Setup

Flannel — Built-In, Nothing to Do

# Flannel ships with K3s — just install
curl -sfL https://get.k3s.io | sh -

# Change backend in /etc/rancher/k3s/config.yaml
flannel-backend: host-gw   # vxlan | host-gw | wireguard-native | none

Installing Calico on K3s

Step 1 — Install K3s without Flannel:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none \
  --disable-network-policy \
  --cluster-cidr=192.168.0.0/16" sh -

Step 2 — Install Calico operator:

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/tigera-operator.yaml

Step 3 — Apply Installation CR:

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    ipPools:
    - cidr: 192.168.0.0/16
      encapsulation: VXLANCrossSubnet
      natOutgoing: Enabled

Installing Cilium on K3s

Step 1 — Install K3s without Flannel:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none \
  --disable-network-policy \
  --disable=servicelb" sh -

Step 2 — Install Cilium via Helm:

helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium \
  --namespace kube-system \
  --set operator.replicas=1 \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=<YOUR_K3S_API_IP> \
  --set k8sServicePort=6443 \
  --set bpf.masquerade=true \
  --set ipam.mode=kubernetes

Minimum Kernel Requirements

Feature	Cilium	Calico eBPF
Basic CNI	≥ 4.9	Any
kube-proxy replacement	≥ 5.2	≥ 5.3
WireGuard encryption	≥ 5.6	≥ 5.6
XDP acceleration	≥ 5.10	≥ 5.10

✅ Ubuntu 22.04 ships kernel 5.15, Debian 12 ships 6.1, Raspberry Pi OS Bookworm ships 6.1 — all satisfy every requirement.

14. Migration Guide on K3s

All migrations follow the same pattern:

drain → clean CNI state → restart K3s with --flannel-backend=none → install new CNI → uncordon

Flannel → Calico

# Step 1: Drain the node
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

# Step 2: Remove Flannel state on the node
systemctl stop k3s
ip link delete flannel.1 2>/dev/null || true
ip link delete cni0 2>/dev/null || true
rm -rf /var/lib/cni /etc/cni/net.d

# Step 3: Set flannel-backend: none in /etc/rancher/k3s/config.yaml, then restart
systemctl start k3s

# Step 4: Install Calico operator
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/tigera-operator.yaml

# Step 5: Uncordon
kubectl uncordon <node>

Flannel → Cilium

# Steps 1–3 same as above (drain, clean, restart with flannel-backend=none)

# Step 4: Install Cilium
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=<API_IP> \
  --set k8sServicePort=6443

# Step 5: Uncordon
kubectl uncordon <node>

💡 Pro Tip: For single-node K3s lab environments, a clean reinstall is always faster and safer than a live migration. Run k3s-uninstall.sh, reinstall with the correct flags, then Helm install your chosen CNI — total time is about 10 minutes.

15. Conclusion

Open-Source CNIs

🟢 Flannel — A masterpiece of minimalism. One job, done perfectly, with near-zero operational overhead. The right choice when simplicity and RAM constraints matter more than policy or observability.
🟠 Calico — The policy-first CNI. BGP-native routing, mature L3/L4 NetworkPolicy, Windows node support, and a pluggable data plane. The right choice when you need robust policy enforcement, prefer no-encap routing, or operate in an environment with existing BGP infrastructure.
🔵 Cilium — The platform CNI. eBPF-native with O(1) service lookup, L7-aware policy with no sidecar, Hubble observability, full pod-traffic encryption, and Cluster Mesh multi-cluster. The most capable networking layer available in Kubernetes today.
🟡 Weave Net — Once a popular choice for simplicity and built-in encryption. Now archived — migrate to Cilium or Calico for any new or long-running cluster.
🟣 Antrea — The VMware-native CNI. OVS dataplane, tiered policy, Windows support, and NSX-T integration. The right choice in Tanzu or NSX environments.
🔷 Multus — Not a CNI replacement but a CNI multiplier. Essential for telco/NFV workloads needing multiple pod network interfaces.

Cloud Provider CNIs

🔶 AWS VPC CNI (EKS) — Native VPC IP assignment with no overlay. Pods are first-class VPC citizens. Add Calico or the AWS-native policy controller for NetworkPolicy. Choose prefix delegation for high pod density.
🔷 Azure CNI (AKS) — Use Azure CNI Overlay for most production workloads to avoid IP exhaustion, and add the Cilium dataplane for eBPF policy + Hubble observability. Azure CNI traditional still works, but requires careful subnet pre-planning.
♦️ GKE Dataplane V2 (GKE) — Google's managed Cilium eBPF layer. The default for new GKE clusters. Handles NetworkPolicy at scale with eBPF O(1) lookups. Add the Hubble observability add-on for network telemetry. Self-manage Cilium on GKE only if you need L7 CNP or Cluster Mesh.

Bottom line: If you run a managed Kubernetes service, use the cloud-default CNI and layer policy/observability on top. If you run self-managed clusters, Cilium is the most capable long-term investment, with Calico as the pragmatic choice if BGP integration or Windows nodes are required.

The networking layer of your cluster is not where you want to cut corners at scale.
Choose based on where your cluster is going — not just where it is today.

🔴 Supply Chain Attacks Are Breaking the Internet in 2026 — Every Major Hack Explained

Pendela BhargavaSai — Tue, 05 May 2026 04:00:00 +0000

Your vulnerability scanner is hacking you. Your password manager got weaponized. Your AI coding tool is the new attack surface. Welcome to 2026.

The Year Everything Became a Weapon

In 2025, supply chain attacks were a concern. In 2026, they became the dominant threat vector in software security.

The numbers are staggering: a single compromised maintainer account poisoned a library with 100 million weekly downloads. A misconfigured CI/CD workflow cascaded into five separate tool compromises within days. A developer downloaded Roblox exploit scripts, and that mistake eventually exposed Vercel's internal database — which was listed for sale at $2 million on BreachForums.

This isn't theoretical risk. This is what happened between January and April 2026.

In this post, I'm going to break down every major supply chain attack that hit the IT and software ecosystem this year — what got compromised, how the attackers did it, what the real blast radius looked like, and most importantly, what you need to do right now to protect your pipelines.

Let's start with what a supply chain attack actually is — because most explanations bury the lead.

What Is a Software Supply Chain Attack?

Here's the mental model that matters:

Instead of breaking into your house, the attacker bribes your locksmith.

When you run npm install or pip install, you're implicitly trusting thousands of strangers who maintain open-source packages. You're trusting their accounts, their CI/CD pipelines, their GitHub credentials, and their judgment. Every single one of those trust relationships is an attack surface.

A supply chain attack exploits that trust. Instead of targeting you directly — which requires defeating your firewall, your endpoint detection, your access controls — attackers target the supplier. Compromise one maintainer account, and you've just compromised every developer who installs that package.

The attack chain looks like this:

1. `Identify` a maintainer of a widely-used package
2. Phish their npm/GitHub credentials, or exploit a misconfigured CI/CD workflow
3. Push backdoored versions — the malware runs at install time or on startup
4. Harvest: cloud credentials, SSH keys, API tokens, Kubernetes configs
5. Cascade: use stolen tokens to compromise more repos, more pipelines, more packages
6. Monetize: ransomware, data sale on BreachForums, cryptomining

The asymmetry is what makes this so devastating. The attacker breaks in once, at one point in the supply chain, and inherits access to thousands of downstream organizations simultaneously.

Now let's talk about what actually happened in 2026.

January 2026 — Cisco Unified Communications Zero-Day (CVE-2026-20045)

What got compromised

Cisco's entire enterprise voice stack: Unified Communications Manager, IM & Presence Service, Unity Connection, and Webex Calling Dedicated Instance.

How it happened

A critical zero-day in the web-based management interface allowed unauthenticated remote attackers to send crafted HTTP requests and execute arbitrary commands on the underlying OS — then escalate straight to root. No credentials needed. No user interaction required.

Why it's a supply chain risk (not just a vulnerability)

This one is subtler than the package ecosystem attacks below, but it's a textbook supply chain risk: managed service providers. Thousands of organizations outsource their voice and UC infrastructure to third parties. If your managed service provider is running vulnerable Cisco UC components, your business communications become a pivot point into your environment — even if your own perimeter is airtight.

This is the definition of inherited risk. You didn't deploy the vulnerable software. You didn't configure it. But you're exposed because you trusted someone who did.

How to protect yourself

Apply Cisco's emergency patch immediately (see Cisco Security Advisory cisco-sa-20260115-uc)
Implement continuous vendor monitoring — when a critical advisory drops, you need instant visibility into which of your vendors is exposed
Restrict management interface access to known IP ranges only
Map which applications and data flows depend on your vendors' UC components so you can assess blast radius before an attack, not after

February 2026 — GitHub Actions: The Misconfiguration That Started Everything

What got compromised

This is the origin point of the largest multi-tool supply chain campaign of 2026. A threat actor operating under the GitHub handle hackerbot-claw (account created February 20, 2026) ran an automated campaign scanning public repositories for a specific GitHub Actions misconfiguration: the pull_request_target event trigger with excessive token permissions.

On February 27–28, the attacker successfully exploited this misconfiguration in Aqua Security's Trivy repository, exfiltrating the aqua-bot service account's Personal Access Token (PAT). This PAT had write access to release automation — which is everything the attacker needed.

How it happened

The pull_request_target workflow is a GitHub Actions feature that lets CI pipelines trigger automatically on pull requests from external contributors. The problem: when misconfigured, external code gets access to the repository's internal secrets. The workflow essentially hands an untrusted contributor the keys to your pipeline.

Aqua detected the intrusion and attempted credential rotation. But here's the critical failure: the rotation was not atomic. Sequential token replacement left a window during which newly issued tokens may have been captured. As Aqua's VP of Open Source, Itay Shakury, later confirmed:

"We rotated secrets and tokens, but the process wasn't atomic, and attackers may have been privy to refreshed tokens."

This residual access enabled everything that followed in March.

The lesson about `pull_request_target`

This is a well-documented dangerous pattern, but it keeps getting deployed:

# ⚠️ DANGEROUS — external PRs can access your secrets
on:
  pull_request_target:
    types: [opened]

jobs:
  ci:
    permissions:
      contents: write  # ← This is the mistake

# ✅ SAFE — pin to SHA, restrict permissions
on:
  pull_request:

jobs:
  ci:
    permissions:
      contents: read  # minimum required

How to protect yourself

Never use pull_request_target with write permissions for workflows triggered by external contributors
Pin all GitHub Actions to full 40-character commit SHAs — not version tags (more on why this matters below)
Rotate credentials atomically — revoke all, reissue all, in a single synchronized operation
Limit service account tokens to minimum required permissions and scope

March 2026 — The Month Everything Went Wrong

March 2026 will go down as the most significant month in software supply chain history. Five major compromises. One threat group. A cascade that went from a misconfigured GitHub workflow to a ransomware operation targeting 1,000+ enterprise SaaS environments.

Let me break each one down.

🔍 Trivy (Aqua Security) — March 19–20, 2026

CVE-2026-33634 | Severity: CRITICAL

What happened

At approximately 17:43 UTC on March 19, 2026, an attacker with residual access from the February compromise force-pushed malicious code to 75 of 77 version tags in aquasecurity/trivy-action — the official GitHub Action for Trivy, one of the most widely deployed open-source vulnerability scanners in the world.

Simultaneously, all 7 tags in aquasecurity/setup-trivy were poisoned, and a weaponized Trivy binary (v0.69.4) was published to GitHub Releases, Docker Hub, GHCR, ECR Public, and deb/rpm repositories.

Safe versions: only trivy-action v0.35.0, setup-trivy v0.2.6, and trivy v0.69.3 were unaffected.

The attack was elegant and terrifying

The malicious entrypoint.sh ran the credential-harvesting payload first, then ran the legitimate Trivy scan. Workflows completed normally. No errors. No indication of compromise. Developers watching their CI logs saw a clean vulnerability scan — while their secrets were being exfiltrated in the background.

The malware (named "TeamPCP Cloud Stealer") performed three operations:

Dumped Runner.Worker process memory to extract GitHub PATs and CI secrets
Swept SSH keys, cloud credentials (AWS, GCP, Azure), Kubernetes tokens, Docker configs, Git credentials
Encrypted the bundle with AES-256 + RSA-4096 and exfiltrated to attacker-controlled servers

If the primary C2 channel failed, the malware fell back to creating a repository called tpcp-docs inside the victim's own GitHub organization to store stolen secrets. Check your org for that repo right now.

The forensic tells (that most teams missed)

# Each malicious commit had an impossible timestamp:
# - Claimed to be from 2021/2022
# - But parent commit was dated March 2026

# Additionally:
# - Only entrypoint.sh was modified per commit
# - Original commits touched multiple files
# - GitHub's "Immutable" release badge was present (but meaningless)

How to protect yourself

# ❌ VULNERABLE — tag can be rewritten silently
- uses: aquasecurity/trivy-action@v0.34.2

# ✅ SECURE — commit SHA is immutable
- uses: aquasecurity/trivy-action@f781cce5aab226378d021711787766a7d423d18d

If you ran Trivy between 17:43 and 23:13 UTC on March 19, 2026:

Search your GitHub org for any repo named tpcp-docs
Check DNS/network logs for connections to scan.aquasecurtiy[.]org (note the typo — deliberate)
Check for connections to 45.148.10.212
Treat all CI/CD secrets from that window as fully compromised — rotate everything

🧠 LiteLLM — March 24, 2026

Severity: CRITICAL | ~3.4M daily downloads | 40-minute exposure window

What happened

LiteLLM is a Python package providing a unified interface for 100+ LLM APIs — OpenAI, Anthropic, Google, AWS Bedrock, Azure OpenAI. Because it sits between your applications and multiple AI providers, it has access to API keys and cloud credentials for all of them. That's exactly why it was targeted.

The compromise was a cascade from Trivy: LiteLLM's CI/CD pipeline used Trivy for security scanning. When Trivy was poisoned on March 19, the malware in LiteLLM's pipeline exfiltrated its PyPI publish token to TeamPCP. Five days later, attackers used that token to upload two malicious versions directly to PyPI:

litellm==1.82.7 — published 10:39 UTC
litellm==1.82.8 — published 10:52 UTC

Both were live for approximately 40 minutes before PyPI quarantined them. During that window, they accumulated tens of thousands of downloads.

The 3-stage payload

# Stage 1: Credential Harvesting
# Exfiltrates to models.litellm.cloud (attacker-controlled, not official BerriAI domain)
collect([
    "LLM API keys (OpenAI, Anthropic, Google...)",
    "Cloud credentials (AWS, GCP, Azure)",
    "SSH keys, shell history, .env files",
    "Crypto wallets",
    "Kubernetes configs"
])

# Stage 2: Kubernetes Lateral Movement
# Deploys privileged DaemonSets → full cluster access

# Stage 3: Persistence
# Installs ~/.config/systemd/user/sysmon.service
# Polls attacker server for additional payloads
# Survives package removal

The .pth file mechanism in v1.82.8 was particularly nasty: it placed a litellm_init.pth file that executed on every Python interpreter startup — meaning the payload fired even when LiteLLM wasn't explicitly imported.

Disclosure suppression

When the community opened GitHub issue #24512 to report the compromise, TeamPCP deployed 88 bots from 73 unique compromised developer accounts in a 102-second window to spam the thread. They used the compromised maintainer account to close the issue as "not planned." This is one of the first documented uses of AI-assisted bot networks for supply chain attack disclosure suppression.

Immediate action

# Check if you're affected
pip show litellm | grep Version
# v1.82.7 or v1.82.8 = COMPROMISED

# Check for persistence
ls ~/.config/systemd/user/sysmon.service
ls ~/.config/sysmon/sysmon.py

# In Kubernetes
kubectl get pods -n kube-system | grep "node-setup"

# Purge cache
pip cache purge
# or
rm -rf ~/.cache/uv

# Safe version
pip install litellm==1.82.6

📦 Axios (npm) — March 30–31, 2026

Severity: CRITICAL | ~100M weekly downloads | Attributed: UNC1069 (North Korea)

What happened

Axios is one of the most depended-upon libraries in the JavaScript ecosystem. At the time of the attack, it was present in approximately 80% of cloud and code environments. The attack didn't exploit any code vulnerability — it was a straightforward account takeover.

Attackers compromised the npm account of jasonsaayman, Axios's primary maintainer, by changing the account's associated email from jasonsaayman@gmail.com to ifstap@proton.me. This bypassed the GitHub Actions OIDC publish flow entirely.

The attack timeline:

2026-03-30 05:57 UTC — plain-crypto-js@4.2.0 published (clean decoy, builds registry history)
2026-03-30 23:59 UTC — plain-crypto-js@4.2.1 published (malicious postinstall backdoor)
2026-03-31 00:21 UTC — axios@1.14.1 published (MALICIOUS, tagged: latest)
2026-03-31 01:00 UTC — axios@0.30.4 published (MALICIOUS, tagged: legacy)
2026-03-31 03:29 UTC — Detected and removed

39 minutes. Two malicious versions. Both tagged as the default install.

The payload

The malicious dependency plain-crypto-js contained a postinstall hook that silently downloaded and executed platform-specific stage-2 RAT implants from sfrclak[.]com:8000. Cross-platform: macOS, Windows, Linux.

Google's Threat Intelligence Group attributed this to UNC1069, a financially motivated North Korean threat actor. OpenAI was sufficiently exposed via Axios's dependency chain that it revoked its macOS code-signing certificate on March 31, 2026 as a precaution.

Check your lockfiles now

# Check for compromised versions
grep -E "axios.*(1\.14\.1|0\.30\.4)" package-lock.json
grep -E "plain-crypto-js" package-lock.json yarn.lock bun.lockb

# Safe versions
npm install axios@1.14.0  # Last legitimate 1.x with SLSA provenance

How to protect yourself

Enable phishing-resistant MFA on npm, GitHub, and all cloud platforms — no exceptions
Use npm ci with strict lockfiles instead of npm install
Monitor npm for maintainer email changes on critical dependencies
Audit and block postinstall scripts in CI environments where possible
Never run npm install on production systems from ephemeral runners without lockfile pinning

🤖 Anthropic Claude Code — March 31, 2026

Severity: HIGH | ~512,000 lines of proprietary source code | Root cause: Human error

What happened

This one is different from the others — it wasn't a malicious actor compromising a third party. Anthropic accidentally shipped the entire source code of Claude Code to the public npm registry.

When Anthropic published @anthropic-ai/claude-code version 2.1.88, a missing exclusion rule in the build configuration caused a 59.8 MB JavaScript source map file (cli.js.map) to be bundled into the package. That source map pointed to a zip archive on Anthropic's Cloudflare R2 storage containing the full, unobfuscated TypeScript source — 512,000 lines across 1,906 files.

Security researcher Chaofan Shou spotted it on X within hours. By the time Anthropic pulled the package at ~08:00 UTC, the code had been downloaded from their own cloud storage, mirrored to GitHub, and forked tens of thousands of times.

What was exposed

Complete multi-agent orchestration architecture
Self-healing memory system (MEMORY.md architecture with lazy-load topic files)
"Undercover Mode" — suppresses Anthropic-internal metadata in commits to public repos
Anti-distillation controls — injects fake tool definitions into API responses to poison competitor training data
44 feature flags, including an unreleased Tamagotchi easter egg planned for April 1–7
Bidirectional CLI-to-IDE communication layer

The cascading danger

The leak coincided — entirely coincidentally — with the Axios RAT attack. Anyone who updated Claude Code via npm between 00:21 and 03:29 UTC on March 31 may have simultaneously pulled a trojanized version of Axios.

Additionally, attackers immediately registered npm packages mimicking Anthropic's internal tooling (audio-capture-napi, color-diff-napi, image-processor-napi) to stage dependency confusion attacks against developers trying to compile the leaked source.

Do not download, fork, build, or run any GitHub repository claiming to be "leaked Claude Code." Many of these repositories are active malware lures delivering Vidar Stealer and GhostSocks.

Anthropic's official statement

"Earlier today, a Claude Code release included some internal source code. No sensitive customer data or credentials were involved or exposed. This was a release packaging issue caused by human error, not a security breach. We're rolling out measures to prevent this from happening again."
— Anthropic Spokesperson

What this means for your build pipeline

# The failure point: Bun generates source maps by default.
# A single missing line in build config exposed 512K lines of IP.

# Lesson: Add this to your CI/CD pre-publish checklist:
✓ Verify .npmignore excludes *.map files
✓ Verify `files` field in package.json is allowlist-based, not denylist
✓ Run `npm pack --dry-run` and inspect the manifest before every publish
✓ Set up automated secret/source scanning on all npm publish workflows

April 2026 — The Attacks Keep Coming

▲ Vercel — April 19, 2026

Severity: CRITICAL | Entry point: AI productivity tool | Dwell time: ~2 months

What happened

This attack is a masterclass in how OAuth trust relationships create invisible lateral movement paths.

The chain:

February 2026: A Context.ai employee downloaded Roblox game exploit scripts. Those scripts installed Lumma Stealer malware.
Lumma Stealer exfiltrated the employee's Google Workspace OAuth tokens.
Context.ai's Chrome Extension had been granted full Google Drive read access by users during onboarding.
A Vercel enterprise employee had used Context.ai and connected their Vercel Google account.
Attackers pivoted from the stolen tokens → Context.ai's AWS environment → OAuth tokens for their product → the Vercel employee's workspace → Vercel's internal systems.

Vercel disclosed the breach on April 19, 2026. By then, the attacker had approximately 2 months of dwell time. Vercel's CEO Guillermo Rauch confirmed the attack chain publicly on X and named Context.ai as the compromised third party.

The stolen Vercel internal database was listed for sale at $2 million on BreachForums by ShinyHunters.

The env variable problem

Vercel's environment variable model left variables not explicitly marked as "sensitive" unencrypted at rest. Once an attacker had team-scoped OAuth access, they could read all non-sensitive environment variables — connection strings, API keys, third-party service credentials — stored by developers who assumed they were protected.

Key takeaway for developers

You can have perfect security in your own systems and still get breached because an AI productivity tool you gave full Drive access to got compromised via an employee who downloaded Roblox scripts.

This is the supply chain threat model in its purest form. The attack surface is no longer just your code — it's every OAuth permission you've ever granted.

How to protect yourself

Immediate actions:
✓ Audit all OAuth app permissions in your Google Workspace — revoke apps with excessive access
✓ Mark ALL Vercel environment variables as "sensitive" explicitly (not just secrets)
✓ Query database connection logs for IPs outside known egress ranges, Feb–Apr 2026 window
✓ Rotate all API keys and secrets stored in Vercel project environment variables

Systemic changes:
✓ Never grant AI tools full-read workspace access — use scoped permissions
✓ Implement OAuth token monitoring to detect abnormal access patterns
✓ Treat third-party AI tools with the same vendor risk assessment as any SaaS platform

🔐 Bitwarden CLI — April 22, 2026

Severity: CRITICAL | Window: 90 minutes | Notable: First supply chain attack targeting AI coding tools

What happened

The Shai-Hulud worm's "Third Coming." At 5:57 PM ET on April 22, 2026, attackers published @bitwarden/cli@2026.4.0 — a malicious version of the CLI tool for the world's most popular open-source password manager (10M+ users, 50,000 business customers). By 7:30 PM ET, it was gone.

90 minutes. That's the entire attack window.

The attack vector: Bitwarden's repository uses checkmarx/ast-github-action — one of the GitHub Actions compromised in the ongoing Checkmarx supply chain campaign (also attributed to TeamPCP). Attackers hijacked Bitwarden's CI/CD pipeline, editing the publish-cli.yml workflow five consecutive times to inject a prebuilt malicious tarball containing the payload bw1.js.

Bitwarden confirmed: no user vault data was accessed. The web extension, desktop apps, and all other clients were unaffected. Only the CLI npm package was compromised.

The payload was remarkable

The malware targeted six distinct credential surfaces and introduced two novel capabilities:

// Credential targets:
targets = [
  "AWS access keys + SSM/Secrets Manager",
  "Azure credentials + Key Vault",
  "GCP service account keys + Secret Manager",
  "GitHub PATs + npm publish tokens",
  "SSH keys + shell history + .env files",
  "AI coding assistant configurations"  // ← NEW in 2026
]

// Novel capability 1: AI tool targeting
// Explicitly probed for: Claude, Cursor, Codex CLI, Aider
// If authenticated session found → extract credentials + inject persistence

// Novel capability 2: Self-propagating worm
// Uses victim's npm publish tokens to backdoor ALL packages they can publish to
// Exfiltrates to public GitHub repos (RSA-encrypted) as dead-drop C2
// GitHub traffic not flagged by security tools → effective evasion

// Kill switch: skips if Russian locale detected

This changes the threat model for AI coding tools

The Bitwarden CLI attack — combined with the Vercel breach via Context.ai — confirms a clear pattern that security teams need to internalize:

AI coding tools (Claude, Cursor, Copilot, Aider) sit at the intersection of everything attackers want: source code access, command execution, API credentials, and cloud service connections.

These tools are now explicitly named in supply chain attack malware. Your AI coding assistant's authentication state is a credential worth stealing.

Immediate response

# Check if affected (installed between 5:57–7:30 PM ET, April 22)
npm list @bitwarden/cli  # 2026.4.0 = COMPROMISED

# Clean install
npm uninstall -g @bitwarden/cli
npm cache clean --force
npm install -g @bitwarden/cli@2026.4.1  # verified clean

# Find C2 artifacts
find / -name "bw1.js" -o -name "bw_setup.js" 2>/dev/null

# Search for data exfil repos
# Check public GitHub for repos containing: "Shai-Hulud: The Third Coming"

# Rotate if affected:
# → GitHub PATs
# → npm tokens  
# → AWS access keys
# → GCP service account keys
# → Azure credentials
# → SSH keys

The Big Picture: TeamPCP and the Campaign Architecture

Most of the March–April attacks trace back to a single threat group: TeamPCP (also operating as DeadCatx3, PCPcat, Persy_PCP, ShellForce, and CipherForce).

TeamPCP first appeared in late December 2025 as a group focused on cloud-native infrastructure exploitation. Their 2026 campaign was methodical:

Phase 1 (Feb 27–28):  Exploit pull_request_target in Trivy → steal aqua-bot PAT
Phase 2 (Mar 1):      Aqua rotates credentials → incomplete rotation
Phase 3 (Mar 19):     Use residual access → poison 75 Trivy tags + Docker images
Phase 4 (Mar 21):     Use stolen PATs from Trivy → poison KICS GitHub Actions
Phase 5 (Mar 24):     Use LiteLLM CI's Trivy → steal PyPI token → poison LiteLLM
Phase 6 (Mar 27):     Telnyx Python SDK compromised
Phase 7 (Mar 30–31):  Axios npm package poisoned (separate North Korean actor)
Phase 8 (Apr 15):     Vect ransomware lists first victim from Trivy campaign
Phase 9 (Apr 22):     Bitwarden CLI poisoned via Checkmarx GitHub Action

The campaign spanned PyPI, npm, Docker Hub, GitHub Actions, and OpenVSX in a single coordinated multi-ecosystem operation.

Systemic Defenses: What Actually Works

After cataloging all of this, here's what the evidence shows actually works:

1. Pin to commit SHAs, not version tags

# This is the single highest-impact change you can make:

# ❌ VULNERABLE (both of these)
uses: some-action@v2.0
uses: some-action@main

# ✅ IMMUTABLE — cannot be silently changed
uses: some-action@a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2

The 2025 tj-actions attack and the 2026 Trivy attack both succeeded because developers referenced actions by tag. Both would have been completely immune with SHA pinning. One line of config change. That's it.

2. Use lockfiles strictly

# In CI/CD pipelines:
npm ci          # NOT npm install
pip install --require-hashes -r requirements.txt

# Never allow unpinned transitive dependencies in production

3. Atomic credential rotation

When you detect a compromise and rotate credentials, the rotation must be a single synchronized operation — revoke all active tokens, generate new ones, update all consumers simultaneously. Sequential rotation leaves a window. TeamPCP exploited exactly this window in the Trivy incident.

4. Principle of least privilege for service accounts

# Your CI service account should not have:
# - write access to multiple repositories
# - admin access to package registries
# - broad cloud IAM roles

# It should have exactly:
# - read access to the specific repos needed for this job
# - publish access to the specific package this job publishes
# - no persistent credentials (use OIDC/short-lived tokens)

5. Behavior-based CI monitoring

The LiteLLM incident was caught first by a developer whose machine started stuttering — their CPU was pegged because the malware's fork bomb behavior crashed the system. That's not monitoring; that's luck.

What you actually need: alerts for Python processes making outbound POST requests at install time. Package installation should pull from PyPI — it should never POST encrypted binary payloads to external endpoints.

Alert rule: 
  process: python (via pip subprocess)
  direction: outbound
  method: POST
  payload: encrypted binary
  action: ALERT + BLOCK

6. Audit OAuth permissions regularly

The Vercel breach started with a productivity tool that was granted full Google Drive read access. Every OAuth integration in your organization is a potential pivot point. Audit them. Scope them to minimum required permissions. Revoke anything that hasn't been used recently.

7. Treat your AI coding tools as high-privilege systems

Given the Bitwarden CLI attack explicitly targeted Claude, Cursor, Codex, and Aider credentials, it's time to treat AI coding assistant authentication state with the same security posture as cloud access keys:

Don't leave AI tools authenticated in unmonitored environments
Rotate AI tool API keys on the same schedule as cloud credentials
Monitor for abnormal AI tool usage patterns (large data transfers, unusual API calls)
Be aware that your AI coding assistant may have access to your entire codebase, your git credentials, and your cloud service connections simultaneously

The Developer's Quick Reference Checklist

Here's a condensed action list you can use right now:

For your CI/CD pipelines

☐ Pin all GitHub Actions to full commit SHAs
☐ Use npm ci / pip install --require-hashes (not npm install / pip install)  
☐ Audit pull_request_target workflows for excessive permissions
☐ Limit service account tokens to minimum required scope
☐ Enable GitHub's SHA pinning organizational policy
☐ Set up behavior-based alerts for install-time network requests

For your dependencies

☐ Check for plain-crypto-js in any lockfile (Axios RAT indicator)
☐ Check for litellm==1.82.7 or 1.82.8 in any Python environment
☐ Check @bitwarden/cli for version 2026.4.0 (rotate if found)
☐ Search your GitHub org for repos named "tpcp-docs" (Trivy compromise indicator)
☐ Audit all GitHub Actions for recent unexpected workflow edits

For your organization

☐ Audit all OAuth app permissions — revoke excessive access
☐ Mark all Vercel environment variables as "sensitive"
☐ Rotate credentials from any CI pipeline that ran Trivy on March 19, 2026
☐ Implement vendor monitoring with automated CVE-to-vendor mapping
☐ Document your complete dependency tree (can you answer: what packages did production install in the last 30 days?)

Closing Thoughts

The pattern across all of these attacks is the same: attackers are targeting trust, not systems.

They're not breaking through your firewall. They're getting invited through the front door — via a trusted package, a trusted OAuth app, a trusted GitHub Action, a trusted vulnerability scanner.

The question isn't whether your perimeter is secure. The question is whether you know every entity you trust, what access you've granted them, and what happens to your environment if any one of them is compromised.

Supply chain security in 2026 isn't a specialized discipline anymore. It's table stakes for any team that ships software.

The next compromised package is already on its way to your CI pipeline. The question is whether you'll see it land.

Resources and Further Reading

If you're responsible for a CI/CD pipeline, share this with your team — the SHA pinning point alone is worth the read.*

All technical details sourced from public security disclosures, vendor incident reports, and independent researcher analysis.

Tags: #security #cybersecurity #devops #opensource #supplychain #javascript #python #npm #github

The Definitive Guide to Lightweight Kubernetes: KIND, Minikube, MicroK8s, K3s, Vcluster, k0s, and RKE2 Compared

Pendela BhargavaSai — Thu, 23 Apr 2026 03:18:00 +0000

TL;DR — There is no single "best" lightweight Kubernetes. KIND wins CI/CD, Minikube wins local dev UX, MicroK8s wins on Ubuntu, K3s wins edge and production, Vcluster wins multi-tenancy, k0s wins zero-dependency ops, and RKE2 wins enterprise compliance. This post explains why — with architecture diagrams, feature tables, and real-world guidance.

Why Lightweight Kubernetes Matters
The Contenders at a Glance
KIND — Kubernetes IN Docker
Minikube — The Developer's Workhorse
MicroK8s — Zero-Ops by Canonical
K3s — Production-Grade at the Edge
Vcluster — Kubernetes Inside Kubernetes
k0s — Zero Dependencies, Zero Friction
RKE2 — Security-First Enterprise K8s
Scoring Across 8 Dimensions
Use Case Decision Guide
Final Verdict

Why Lightweight Kubernetes Matters

Full-fat Kubernetes — the kind you run on a 3-master, 6-worker production cluster — is extraordinary infrastructure. It is also deeply impractical when you need to:

Spin up a throwaway cluster in a GitHub Actions runner in under 30 seconds
Run Kubernetes on a Raspberry Pi with 1 GB of RAM
Give every developer on your team their own isolated cluster without buying new hardware
Deploy to a factory floor where the "server" is an ARM SBC with no internet access

The Kubernetes ecosystem responded by producing a rich family of lightweight distributions, each making different trade-offs. By 2025, the major players are KIND, Minikube, MicroK8s, K3s, Vcluster, k0s, and RKE2 — and choosing between them is genuinely consequential.

This guide gives you the full picture: architecture, components, features, limitations, scoring, and concrete use-case guidance, all in one place.

The Contenders at a Glance

Tool	Creator	Year	Primary Use Case	Min RAM	Binary Size
KIND	Kubernetes SIG Testing	2019	CI/CD testing	2 GB	N/A (uses Docker)
Minikube	Kubernetes Community	2016	Local development	2 GB	~100 MB
MicroK8s	Canonical (Ubuntu)	2018	Ubuntu / Edge	540 MB	Snap package
K3s	Rancher Labs (SUSE)	2019	Edge / Production	512 MB	< 100 MB
Vcluster	Loft Labs	2021	Multi-tenancy	Host-dependent	Helm chart
k0s	Mirantis	2020	Zero-dependency ops	1 GB	~230 MB
RKE2	Rancher (SUSE)	2021	Enterprise / Compliance	4 GB	~300 MB

Each of these is CNCF-compatible and capable of running real Kubernetes workloads. The differences are in where, how, and at what cost they do it.

1. KIND — Kubernetes IN Docker

What It Is

KIND (Kubernetes IN Docker) was built by the Kubernetes SIG Testing team for one purpose: to test Kubernetes itself. Every node in a KIND cluster is a Docker container. The control plane runs in one container, worker nodes in others, and they communicate over a Docker bridge network called kindnet.

KIND runs every Kubernetes node as a Docker container. There is no VM, no hypervisor, no separate OS. The kindnet CNI is a purpose-built bridge that understands this container-as-node topology. The practical effect is that KIND clusters are disposable, fast, and completely ephemeral — perfect for testing but incapable of persistence.

Because there is no VM involved, KIND clusters start in about 30 seconds and use only Docker's existing networking and storage. You can run a dozen isolated clusters on a single laptop.

Architecture


┌─────────────────────────────────────────────────────---┐
│                   Docker Host                          │
│                                                        │
│  ┌─────────────────────┐   ┌──────────────────────┐    │
│  │   Control Plane     │   │     Worker 1         │    │
│  │   (container)       │──▶│     (container)      │   │
│  │                     │   │                      │    │
│  │  • API Server       │   │  • kubelet           │    │
│  │  • etcd             │   │  • kube-proxy        │    │
│  │  • Scheduler        │──▶│  • Pod A  • Pod B    │    │
│  │  • Controller Mgr   │   └──────────────────────┘    │
│  │  • kindnet CNI      │     ┌──────────────────────┐  │
│  └─────────────────────┘     │     Worker 2         │  │
│                              │     (container)      │  │
│  ┌──────────────────┐        │  • kubelet + pods    │  │
│  │  Port-forwarding │        └──────────────────────┘  │
│  │  localhost:6443  │                                  │
│  └──────────────────┘                                  │
└─────────────────────────────────────────────────────---┘

Core Components

kindnet — Custom CNI using a kernel bridge, purpose-built for KIND's container-as-node model
etcd — Full etcd running inside the control-plane container
containerd — Container runtime inside each node-container (Docker-in-Docker)
kubeadm — KIND uses kubeadm internally to bootstrap the cluster

Key Features

True multi-node clusters (control plane + N workers) on a single host
Custom node images — test against any Kubernetes version
Rootless mode via rootless Docker/Podman
IPv6 and dual-stack support
Create multiple isolated clusters simultaneously
Parallel cluster creation
KUBECONFIG auto-export
Optimised for GitHub Actions, GitLab CI, and Jenkins

Quick Start


# Install

curl  -Lo  ./kind  <https://kind.sigs.k8s.io/dl/v0.22.0/kind-linux-amd64>

chmod  +x  ./kind && sudo  mv  ./kind  /usr/local/bin/kind

# Create a single-node cluster

kind  create  cluster

# Create a multi-node cluster

cat <<EOF | kind  create  cluster  --config=-

kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

nodes:

- role: control-plane

- role: worker

- role: worker

EOF

# Delete cluster

kind  delete  cluster

Pros

Blazing fast — 30-second cluster creation, no hypervisor boot time
Zero VM overhead — runs entirely inside Docker containers
True multi-node topology on one host
Exact Kubernetes version control via node images
Perfect for ephemeral CI environments
No LoadBalancer hacks needed for testing (use NodePort)
Widely supported in CI platforms

Cons

Requires Docker or Podman to be running
Not production-ready under any circumstances
No GPU passthrough
LoadBalancer type services need MetalLB or similar
Volumes are lost when the cluster is deleted
No addon ecosystem

Best For

CI/CD pipelines — specifically integration testing that needs a real multi-node Kubernetes topology without the boot time of a VM-based solution.

2. Minikube - The Developer's Workhorse

What It Is

Minikube is the original local Kubernetes project, released in 2016 and still the most feature-rich local development option. It runs a Kubernetes cluster inside a VM, a container, or directly on the host, and brings an unmatched addon ecosystem of 30+ pre-packaged integrations.

Minikube is the only distribution that abstracts over drivers — it runs identically whether the underlying host is a VM (VirtualBox, HyperKit, KVM), a container (Docker, Podman), or bare metal. This flexibility comes at the cost of startup time and memory, but it means Minikube works for every developer on every operating system.

If you've ever run kubectl apply -f on your laptop, you've probably used Minikube.

Architecture

┌──────────────────────────────────────────────────────────-┐
│             VM / Docker / Podman Driver                   │
│                                                           │
│  ┌─────────────────────────────────────────────────────┐  │
│  │            Single Node (All-in-One)                 │  │
│  │                                                     │  │
│  │  Control Plane                  Data Plane          │  │
│  │  ┌──────────┐  ┌────────────┐   ┌─────────────────┐ │  │
│  │  │API Server│  │etcd        │   │kubelet          │ │  │
│  │  └──────────┘  └────────────┘   │kube-proxy       │ │  │
│  │  ┌──────────┐  ┌────────────┐   │Pod A • Pod B    │ │  │
│  │  │Scheduler │  │Ctrl Manager│   └─────────────────┘ │  │
│  │  └──────────┘  └────────────┘                       │  │
│  └─────────────────────────────────────────────────────┘  │
│                                                           │
│  ┌─────────────────────────────────────────────────────┐  │
│  │                   Addons Layer                      │  │
│  │  Dashboard │ Ingress │ Metrics │ Registry │ Istio   │  │
│  └─────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────-┘

Core Components

Multiple drivers — HyperKit, VirtualBox, KVM2, Docker, Podman, SSH
etcd — Full etcd as the backing store
Calico or Flannel — CNI (configurable per driver)
Addon controller — Manages the 30+ available addon services

Key Features

30+ addons including Istio, Knative, Linkerd, GPU operator, registry, and more
Built-in Kubernetes dashboard (minikube dashboard)
GPU passthrough in VM mode
LoadBalancer via minikube tunnel
Multiple profile management (run several clusters simultaneously)
Image caching to speed up repeated pulls
minikube service command for easy port access
Built-in image loading (minikube image load)

Quick Start


# Install (Linux)

curl  -LO  <https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64>

sudo  install  minikube-linux-amd64  /usr/local/bin/minikube

# Start with Docker driver

minikube  start  --driver=docker

# Enable addons

minikube  addons  enable  ingress

minikube  addons  enable  metrics-server

minikube  addons  enable  dashboard

# Open dashboard

minikube  dashboard

# LoadBalancer support

minikube  tunnel  # Run in separate terminal

# Delete

minikube  delete

Pros

Easiest getting-started experience of any K8s tool
Unmatched addon ecosystem (30+ addons)
GPU passthrough support (VirtualBox/KVM drivers)
Built-in dashboard requires zero configuration
Works on macOS, Linux, and Windows
Multiple profiles = multiple clusters
Best documentation and community support

Cons

Slow startup in VM mode (~2 minutes)
High memory consumption, especially with VM driver
Primarily a single-node environment
Not production-ready
LoadBalancer requires keeping minikube tunnel running separately
Battery-intensive on laptops
Multi-node support exists but is limited and buggy

Best For

Local development — especially developers who want a full Kubernetes experience with addons, dashboards, and GPU support without deep infrastructure expertise.

3. MicroK8s - Zero-Ops by Canonical

What It Is

MicroK8s is Canonical's packaging of Kubernetes as a snap. It installs as a single command, self-heals via systemd, updates automatically through snap channels, and has the lowest memory footprint of any full-featured Kubernetes distribution at just 540 MB.

MicroK8s is unique in using dqlite — a distributed SQLite engine developed by Canonical — as an alternative to etcd for HA mode. This dramatically simplifies the operational burden of running a multi-master cluster: no external etcd cluster needed, just microk8s add-node on each machine.

Unlike KIND and Minikube, MicroK8s is designed for both development and light production workloads. Its HA mode using dqlite (a distributed version of SQLite) supports clustering without requiring a full etcd setup.

Architecture


┌──────────────────────────────────────────────────────────-┐
│                  Snap Package (systemd)                   │
│                                                           │
│  ┌──────────────────────┐   ┌────────────────────────┐    │
│  │    Node 1 (Master)    │   │      Node 2           │    │
│  │                       │   │                       │    │
│  │  • API Server         │──▶│  • kubelet            │    │
│  │  • dqlite (HA store)  │   │  • kube-proxy         │    │
│  │  • Scheduler          │   │  • Calico CNI          │   │
│  │  • Controller Manager │   │  • Pods                │   │
│  │  • Calico CNI         │   └────────────────────────┘   │
│  │  • Auto-updater       │                                │
│  └──────────────────────┘                                 │
│                                                           │
│  ┌──────────────────────────────────────────────────────┐ │
│  │         Addon Engine (microk8s enable <addon>)       │ │
│  │  Istio │ Knative │ GPU │ Registry │ Dashboard │ More │ │
│  └──────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────-┘

Core Components

dqlite — Distributed SQLite for HA without the operational burden of etcd
Calico CNI — Production-grade networking with network policy support
Snap daemon — Manages the entire lifecycle including automatic updates
Addon engine — microk8s enable <name> installs curated addons

Key Features

Lowest memory footprint: 540 MB minimum
HA clustering via microk8s add-node
Automatic channel-based updates with rollback
GPU operator addon for ML/AI workloads
Strict snap confinement for security
ARM64 and x86 native support
Observability stack addon (Prometheus, Grafana)
Built-in image registry

Quick Start


# Install via snap

sudo  snap  install  microk8s  --classic

# Add your user to the microk8s group

sudo  usermod  -aG  microk8s  $USER

newgrp  microk8s

# Check status

microk8s  status  --wait-ready

# Enable core addons

microk8s  enable  dns  ingress  metrics-server  dashboard

# Use kubectl

microk8s  kubectl  get  nodes

# Add worker node (run on master, then copy join command to worker)

microk8s  add-node

# Uninstall

sudo  snap  remove  microk8s

Pros

Lowest RAM usage of all full-featured distributions (540 MB)
Best Ubuntu and Linux integration through the snap ecosystem
Self-healing via systemd — restarts automatically on failure
HA multi-node with a simple add-node workflow
Automatic updates through snap channels (stable, candidate, beta)
Production-capable for light workloads
ARM64 support for Raspberry Pi and ARM servers

Cons

Snap packaging limits portability to non-Ubuntu systems
Ubuntu-centric design — snap is not available everywhere
Addon conflicts can occur (Istio + other service meshes, for example)
Strict snap confinement can block some host filesystem operations
dqlite is still maturing compared to battle-tested etcd
Automatic updates can cause unplanned restarts without configuration

Best For

Ubuntu workstations and edge servers — if you're on Ubuntu, MicroK8s is the most native Kubernetes experience available.

4. K3s - Production-Grade at the Edge

What It Is

K3s is the single most consequential lightweight Kubernetes project of the past five years. Released by Rancher Labs (now SUSE) in 2019, it packs a complete, CNCF-certified Kubernetes distribution into a single binary under 100 MB. It runs on 512 MB of RAM, boots in 30 seconds, and runs identically on a Raspberry Pi, a factory floor ARM controller, and a cloud VM.

K3s achieves its sub-100 MB size by bundling everything into a single Go binary with no external dependencies, using SQLite as a default backing store (which requires no cluster management), and removing upstream K8s features that aren't needed in its target environments (Windows nodes, cloud-provider integrations, certain alpha features).

K3s is not a toy. It is used in production by thousands of organisations worldwide.

Architecture


┌────────────────────────────────────────────────────────────────┐
│                      k3s binary (< 100 MB)                     │
│                                                                │
│  ┌─────────────────────────────────┐                           │
│  │          k3s Server             │                           │
│  │  (Control Plane + Optional DP)  │──────────┐                │
│  │                                 │          │                │
│  │  • API Server                   │          ▼                │
│  │  • SQLite (default) / etcd / PG │   ┌─────────────────┐     │
│  │  • Scheduler                    │   │   k3s Agent 1   │     │
│  │  • Controller Manager           │   │   (Worker Node) │     │
│  │  • Flannel CNI (built-in)       │   │  • kubelet      │     │
│  │  • Traefik Ingress              │   │  • kube-proxy   │     │
│  │  • CoreDNS                      │──▶│  • Flannel     │     │
│  │  • local-path-provisioner       │   │  • Pods         │     │
│  │  • Helm controller              │   └─────────────────┘     │
│  └─────────────────────────────────┘          │                │
│                                               ▼                │
│                                        ┌─────────────────┐     │
│                                        │ k3s Agent 2     │     │
│                                        │ (ARM / IoT)     │     │
│                                        └─────────────────┘     │
└────────────────────────────────────────────────────────────────┘

Core Components

Single binary — Packages containerd, CNI plugins, CoreDNS, Traefik, and more
SQLite — Default data store, ideal for single-server or small clusters
Embedded etcd — Available for HA clusters (3+ servers)
External DB — PostgreSQL, MySQL, or etcd for larger deployments
Flannel CNI — Built-in overlay networking, zero extra configuration
Traefik — Ingress controller included out of the box
Helm controller — Manage Helm charts via CRDs
local-path-provisioner — Dynamic PVC provisioning on local disk

Key Features

CNCF-certified — passes full Kubernetes conformance tests
Single binary < 100 MB with everything bundled
Multiple storage backends: SQLite, etcd, PostgreSQL, MySQL
ARM64 and ARMv7 first-class support
Air-gap / offline install support (critical for edge deployments)
Auto TLS with Let's Encrypt for Traefik
Server + Agent role split for control/data plane separation
Automatic certificate rotation

Quick Start


# Install server (master) — one command

curl  -sfL  <https://get.k3s.io> | sh  -

# Check status

sudo  systemctl  status  k3s

sudo  kubectl  get  nodes

# Get the node join token

sudo  cat  /var/lib/rancher/k3s/server/node-token

# Join a worker node (run on the worker)

curl  -sfL  <https://get.k3s.io> | \\

K3S_URL=https://<SERVER_IP>:6443  \\

K3S_TOKEN=<NODE_TOKEN> \\

sh  -

# Use kubectl without sudo

mkdir  -p  ~/.kube

sudo  cp  /etc/rancher/k3s/k3s.yaml  ~/.kube/config

sudo  chown $(id  -u):$(id  -g) ~/.kube/config

# Uninstall

/usr/local/bin/k3s-uninstall.sh  # server

/usr/local/bin/k3s-agent-uninstall.sh  # agent

HA Setup (Embedded etcd)


# First server node

curl  -sfL  <https://get.k3s.io> | sh  -s  -  server  \\

--cluster-init

# Additional server nodes

curl  -sfL  <https://get.k3s.io> | sh  -s  -  server  \\

--server https://<FIRST_SERVER_IP>:6443 \\

--token <NODE_TOKEN>

Pros

CNCF-certified — genuine, conformant Kubernetes, not a cut-down imitation
Single binary under 100 MB — deploy to anything
512 MB RAM minimum — runs on Raspberry Pi 3
30-second cold start
SQLite for small clusters, etcd for HA — right tool for every scale
Traefik ingress out of the box — production workloads with zero extra config
ARM64 and ARMv7 native — best IoT Kubernetes support in the market
Air-gap install — works in completely offline environments

Cons

SQLite backend not suitable for clusters exceeding ~50 nodes
Some upstream Kubernetes features are stripped (Alpha features, some cloud integrations)
Default CNI is Flannel only (using Calico requires additional configuration)
No built-in dashboard
Less rich addon ecosystem than Minikube or MicroK8s
Limited Windows node support

Best For

Edge computing, IoT, production on resource-constrained hardware, and any environment where the binary size and startup time of a traditional Kubernetes distribution is prohibitive.

5. Vcluster — Kubernetes Inside Kubernetes

What It Is

Vcluster takes a completely different approach to "lightweight Kubernetes." Rather than running alongside a host operating system, it runs inside an existing Kubernetes cluster. Each virtual cluster is a set of pods in a namespace, but from the user's perspective it is a completely isolated Kubernetes cluster with its own API server, etcd, and full Kubernetes API.

This makes Vcluster the definitive answer to the multi-tenancy problem: instead of giving teams namespace isolation (which shares the API server and exposes blast radius), you give each team their own cluster for the cost of a few pods.

Vcluster is architecturally unique in the field. Its virtual control plane (API server + etcd + scheduler + controller manager) runs as pods inside a host cluster namespace. A component called the Syncer watches the virtual cluster's API and translates virtual resources into real host resources — a virtual Pod becomes a real Pod in the host namespace with a remapped name.

Architecture

┌──────────────────────────────────────────────────────────────┐
│               Host Kubernetes Cluster (any provider)         │
│                                                              │
│  ┌────────────────────┐  ┌────────────────────┐              │
│  │    vcluster 1      │  │    vcluster 2       │             │
│  │  (Team A ns)       │  │  (Team B ns)        │             │
│  │                    │  │                     │             │
│  │  Virtual API Srv   │  │  Virtual API Srv    │             │
│  │  In-process etcd   │  │  In-process etcd    │             │
│  │  Syncer pod        │  │  Syncer pod         │             │
│  │  ┌────┐  ┌────┐    │  │  ┌────┐  ┌────┐    │              │
│  │  │PodA│  │PodB│    │  │  │PodC│  │PodD│    │              │
│  │  └─┬──┘  └─┬──┘    │  │  └─┬──┘  └─┬──┘    │              │
│  │    │sync   │sync   │  │    │sync   │sync    │             │
│  └────┼───────┼───────┘  └────┼───────┼────────┘             │
│       ▼       ▼               ▼       ▼                      │
│  ┌──────────────────────────────────────────────────────┐    │
│  │  Shared Worker Nodes — Host CNI, Storage, Hardware   │    │
│  └──────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────┘

The Syncer is the key innovation: it translates virtual cluster resources into real host cluster resources. A Pod created in vcluster 1 becomes a real Pod in the host cluster's namespace, but with a remapped name that prevents conflicts.

Core Components

Virtual API Server — Full Kubernetes API, runs as a pod in the host cluster
In-process etcd — Embedded etcd for the virtual cluster's state
Syncer — Reconciles virtual resources to host cluster resources
vcluster CLI — Manages lifecycle: create, connect, delete, list

Key Features

Full Kubernetes API isolation per virtual cluster
Works on top of any Kubernetes (EKS, GKE, AKS, K3s, RKE2, etc.)
~10 second spin-up time — fastest of all solutions
No extra hardware — uses existing cluster nodes
CRD isolation — each vcluster has its own CRDs
RBAC isolation — separate RBAC per vcluster
Helm chart deployment — deploy via standard Helm
On-demand creation and deletion

Quick Start


# Install vcluster CLI

curl  -L  -o  vcluster  "<https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64>"

chmod  +x  vcluster && sudo  mv  vcluster  /usr/local/bin

# Create a virtual cluster

vcluster  create  my-vcluster  --namespace  team-a

# Connect to it (sets KUBECONFIG automatically)

vcluster  connect  my-vcluster  --namespace  team-a

# Now kubectl talks to the vcluster

kubectl  get  nodes

kubectl  create  deployment  nginx  --image=nginx

# Disconnect

vcluster  disconnect

# Delete

vcluster  delete  my-vcluster  --namespace  team-a

Pros

Full Kubernetes API isolation per tenant — no shared API server blast radius
10-second spin-up — fastest cluster creation of all solutions reviewed
No extra hardware — reuses host cluster's nodes entirely
Works on any cloud or on-premises Kubernetes
Cost-efficient multi-tenancy at scale
Each team gets the full kubectl experience
Easy to create and delete on demand for short-lived environments

Cons

Not standalone — requires a host Kubernetes cluster to exist first
Cannot create real nodes — virtual only
Advanced networking between vclusters is complex
Some cluster-scoped resources (like ClusterRoles and CRDs) are not fully isolated
Requires privileged pod access on the host cluster
Newer project — less battle-tested than K3s or Minikube
Node-level debugging is limited

Best For

Multi-tenant development environments, per-team isolated clusters, and CI/CD environments where many short-lived clusters need to be spun up and torn down rapidly on existing infrastructure.

6. k0s — Zero Dependencies, Zero Friction

What It Is

k0s (pronounced "kay-zeros") from Mirantis lives up to its name: zero host OS dependencies. It is a single binary that includes everything needed to run Kubernetes without requiring any specific kernel modules, swap configuration, or package manager. It works on any Linux distribution out of the box.

k0s uses an eBPF-based CNI called kube-router, includes Autopilot for automated upgrades, and offers FIPS 140-2 compliance — a feature set that appeals strongly to regulated industries.

k0s prioritises deployment universality. By bundling containerd and all CNI plugins into the binary itself and requiring no kernel module configuration from the host OS, it can be dropped onto virtually any Linux system and run. The eBPF-based kube-router CNI offers modern packet processing without iptables overhead.

Architecture


┌──────────────────────────────────────────────────────┐
│             k0s binary (systemd / OpenRC)            │
│                                                      │
│  ┌──────────────────────────┐                        │
│  │     k0s controller       │                        │
│  │   (Control Plane)        │───────────────┐        │
│  │                          │               │        │
│  │  • API Server            │               ▼        │
│  │  • etcd (embedded)       │   ┌─────────────────┐  │
│  │  • Scheduler             │   │  k0s worker 1   │  │
│  │  • Controller Manager    │   │                 │  │
│  │  • containerd            │   │  • kubelet      │  │
│  │  • kube-router (eBPF)    │──▶│  • kube-router  │  │
│  │  • Autopilot updater     │   │  • containerd   │  │
│  └──────────────────────────┘   │  • Pods         │  │
│                                  └─────────────────┘ │
│  k0sctl tool → manages cluster lifecycle             │
└──────────────────────────────────────────────────────┘

Key Features

Truly zero host OS dependencies — no kernel module requirements
FIPS 140-2 compliance mode available
eBPF-based networking via kube-router
Autopilot automated upgrades
k0sctl for full cluster lifecycle management
ARM64 native support
Air-gap install support
Works on any Linux OS (Debian, RHEL, Alpine, CoreOS, etc.)

Quick Start


# Download k0s

curl  -sSLf  <https://get.k0s.sh> | sudo  sh

# Install and start as a service

sudo  k0s  install  controller  --single

sudo  k0s  start

# Get kubeconfig

sudo  k0s  kubeconfig  admin > ~/.kube/config

# Check cluster

kubectl  get  nodes

# Add a worker node — generate join token on controller

sudo  k0s  token  create  --role=worker

# On the worker node

sudo  k0s  install  worker  --token-file  /path/to/token

sudo  k0s  start

Pros

Truly zero host OS dependencies — works on any Linux, no special kernel configuration
FIPS 140-2 compliance for regulated industries
eBPF-based networking with kube-router is modern and efficient
Autopilot handles automated upgrades safely
k0sctl provides a proper cluster lifecycle management tool
No swap or kernel module pre-requirements
Air-gap support

Cons

Smaller community than K3s or MicroK8s
Less rich addon ecosystem
k0sctl adds an additional tool to the workflow
Some CNI plugins need manual configuration beyond kube-router
Enterprise support is a paid product from Mirantis
Fewer third-party integrations and tutorials

Best For

Environments where host OS diversity is a challenge — mixed Linux distributions, heavily locked-down servers, or compliance-driven deployments needing FIPS 140-2.

7. RKE2 — Security-First Enterprise K8s

What It Is

RKE2 (Rancher Kubernetes Engine 2) is the enterprise evolution of K3s. Where K3s optimises for minimal resource usage and edge deployability, RKE2 optimises for security hardening and compliance. It ships hardened by default with CIS Kubernetes Benchmark compliance, FIPS 140-2 support, automatic etcd snapshots, and deep Rancher integration.

RKE2 starts from K3s's architecture and adds a hardening layer: Pod Security Admission enforced by default, etcd encryption at rest, CIS-compliant API server flags, audit logging enabled, and Canal CNI with network policy enforcement. It is Kubernetes made appropriate for government and financial sector requirements.

If K3s is the lightweight sports car, RKE2 is the armoured vehicle. More resource-intensive, harder to damage.

Architecture

┌───────────────────────────────────────────────────────────┐
│              RKE2 Server (Hardened Control Plane)         │
│                                                           │
│  ┌──────────────────────────────────────────────────────┐ │
│  │  CIS-Hardened Kubernetes                             │ │
│  │                                                      │ │
│  │  • Hardened API Server (PSP enforced)                │ │
│  │  • etcd with automated snapshots                     │ │
│  │  • Hardened Scheduler & Controller Manager           │ │
│  │  • Canal / Calico / Cilium CNI (configurable)        │ │
│  │  • containerd runtime                                │ │
│  │  • Cert-manager + auto rotation                      │ │
│  └──────────────────────────────────────────────────────┘ │
│                    │                                      │
│          ┌─────────┴──────────┐                           │
│          ▼                    ▼                           │
│  ┌────────────────┐  ┌────────────────┐                   │
│  │  RKE2 Agent 1  │  │  RKE2 Agent 2  │                   │
│  │  (Worker)      │  │  (Worker)      │                   │
│  └────────────────┘  └────────────────┘                   │
│                                                           │
│  ┌──────────────────────────────────────────────────────┐ │
│  │  Rancher Management Plane (optional)                 │ │
│  └──────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────┘

Key Features

CIS Kubernetes Benchmark v1.6 compliant by default
FIPS 140-2 cryptographic compliance
etcd with automated periodic snapshots and restoration
Multiple CNI options: Canal (default), Calico, Cilium
Automated certificate rotation
Helm chart integration
Air-gap install support
Deep Rancher management platform integration
Role-based node configuration

Quick Start


# Install RKE2 server

curl  -sfL  <https://get.rke2.io> | sh  -

systemctl  enable  rke2-server.service

systemctl  start  rke2-server.service

# Get kubeconfig

export  KUBECONFIG=/etc/rancher/rke2/rke2.yaml

# Get join token for workers

cat  /var/lib/rancher/rke2/server/node-token

# On worker nodes

curl  -sfL  <https://get.rke2.io> | INSTALL_RKE2_TYPE="agent"  sh  -

mkdir  -p  /etc/rancher/rke2/

cat > /etc/rancher/rke2/config.yaml <<EOF

server: https://<SERVER_IP>:9345

token: <NODE_TOKEN>

EOF

systemctl  enable  rke2-agent.service

systemctl  start  rke2-agent.service

Pros

CIS Kubernetes Benchmark compliance out of the box — no manual hardening
FIPS 140-2 for regulated environments (finance, government, healthcare)
Automated etcd snapshots — point-in-time restore capability
Multiple CNI choices (Canal, Calico, Cilium) for varied network requirements
Excellent Rancher multi-cluster management integration
Automated certificate rotation
Strong air-gap support for isolated environments

Cons

4 GB RAM minimum makes it unsuitable for edge/IoT
Longer startup time (~2 minutes)
More operationally complex than K3s
Overkill for non-compliance use cases
Tightly coupled to the Rancher ecosystem
Larger binary and resource footprint
etcd only — no SQLite lightweight option

Best For

Enterprise, compliance-driven, and government workloads where security hardening and audit-readiness are non-negotiable.

Scoring Across 8 Dimensions

Scores are relative (1–10, higher is better for most dimensions):

Dimension	KIND	Minikube	MicroK8s	K3s	Vcluster	k0s	RKE2
Ease of use	7	9	8	7	5	6	4
Production readiness	2	2	7	9	8	8	10
Resource efficiency	7	5	9	10	8	8	3
Multi-node support	9	4	8	9	8	8	9
Addon ecosystem	5	9	8	6	5	4	7
Edge / IoT fit	1	1	6	10	1	7	3
Multi-tenancy	1	1	2	3	10	2	4
CI/CD suitability	10	7	7	7	9	6	5

Use Case Decision Guide

Your Situation	Best Choice	Runner-Up
GitHub Actions / GitLab CI pipelines	KIND	Vcluster
Local development on macOS/Windows/Linux	Minikube	MicroK8s
Developer on Ubuntu workstation	MicroK8s	K3s
Raspberry Pi cluster at home	K3s	MicroK8s
Industrial IoT / factory floor	K3s	k0s
ARM-based edge server	K3s	MicroK8s
Production workload on lightweight infra	K3s	MicroK8s
Government / regulated enterprise	RKE2	k0s
FIPS 140-2 compliance required	RKE2 or k0s	—
Multi-tenant dev environments	Vcluster	Namespace isolation
Per-team isolated clusters	Vcluster	KIND
Mixed Linux OS fleet	k0s	K3s
Air-gap / offline environment	K3s	k0s or RKE2
Testing Kubernetes itself	KIND	—
HA on bare metal with minimal ops	MicroK8s	K3s embedded etcd
Kubernetes with Rancher management	RKE2	K3s

The Decision Tree

Do you need production-grade?
├── No → Is it for CI/CD testing?
│         ├── Yes → KIND
│         └── No  → Are you on Ubuntu?
│                   ├── Yes → MicroK8s
│                   └── No  → Minikube
└── Yes → Do you need compliance (FIPS/CIS)?
          ├── Yes → RKE2 (CIS+FIPS) or k0s (FIPS)
          └── No  → Is it edge/IoT/ARM?
                    ├── Yes → K3s
                    └── No  → Need multi-tenancy?
                              ├── Yes → Vcluster
                              └── No  → K3s or MicroK8s

Final Verdict

After a thorough review, the landscape shakes out clearly:

K3s is the most remarkable project in the lightweight Kubernetes space. It delivers a complete, CNCF-certified Kubernetes distribution in under 100 MB, runs on 512 MB of RAM, and works in air-gapped ARM environments. For the vast majority of production lightweight Kubernetes use cases, K3s is the correct answer.

Vcluster solves a problem no other distribution addresses: genuine Kubernetes API-level multi-tenancy without dedicated hardware. If you need to give 10 teams their own isolated clusters, Vcluster is the only sensible approach.

KIND is indispensable for CI/CD. If you run Kubernetes integration tests in any CI system, KIND's 30-second, Docker-native, multi-node clusters are the right tool with no close competitor.

Minikube remains the best onboarding experience for developers who are new to Kubernetes. The addon ecosystem and built-in dashboard lower the barrier to entry substantially.

MicroK8s is the best Kubernetes for Ubuntu. If your team lives on Ubuntu workstations and servers, snap-based installation, self-healing, and dqlite HA make it the most frictionless operational experience on that platform.

k0s fills an important niche: mixed Linux fleets and environments where zero host OS dependencies matter more than community size or addon richness.

RKE2 is the right answer when your compliance officer needs CIS Kubernetes Benchmark and FIPS 140-2. The resource overhead is the price of admission to heavily regulated sectors.

Resources

This post was written in April 2025. Kubernetes moves fast — always check the official documentation for the latest version information.

Tags: kubernetes k8s k3s kind minikube microk8s vcluster k0s rke2 devops infrastructure edge-computing cloud-native containers cncf

Running k3s on Proxmox: A Multi-Node Cluster with a VM and LXC Worker — The Hard Way and Back

Pendela BhargavaSai — Tue, 21 Apr 2026 03:30:00 +0000

A practical guide covering installation, troubleshooting, and the real story of getting k3s to run inside an LXC container

Introduction

Kubernetes is powerful but notorious for being heavy. k3s, the lightweight Kubernetes distribution from Rancher, fixes that. It strips out legacy APIs, bundles containerd, and ships as a single binary under 100MB. It is perfect for homelabs, edge deployments, and resource-constrained environments.
(more about k3s: https://traefik.io/glossary/k3s-explained/)

This is the first of a series of posts describing how to bootstrap a Kubernetes cluster on Proxmox using ubuntu VM and LXC containers. By the end of the series, the aim is to have a fully working Kubernetes (K3S) install including MetalLB load balancer, Gateway API controller and an Istio service mesh. I’ll also have some sample applications installed for good measure.

Basically why do I need a Kubernetes cluster ?

At work, I’ve used large K8S clusters in production environments (AWS), clusters are abstracted away behind platform teams, which is efficient for delivery but leaves gaps in understanding how scheduling, networking, storage, and controllers really behave under the hood. Setting up your own cluster gives you that missing layer of operational intuition: you get to break things, debug them, and understand why they broke. For someone already running a fairly complex home setup, using Kubernetes as a unifying platform to experiment, whether or not you fully migrate all your Docker Compose stacks—is less about necessity and more about building practical, transferable expertise.

In this post I document how I built a three-node k3s cluster on Proxmox VE with:

1 master node — a Proxmox VM running Ubuntu
1 VM worker node — a standard Proxmox VM (worker1)
1 LXC worker node — a Proxmox LXC container (worker2)

The VM setup was straightforward. The LXC setup was not. This post focuses heavily on the LXC journey — the errors, the fixes, the Linux internals involved, and what it finally took to make it work.

Part 1: Setting Up the Master Node

Installing k3s Server

On the master VM, installing k3s is a single command:


curl  -sfL  https://get.k3s.io | sh  -

k3s sets up a systemd service, installs containerd, and bootstraps a single-node Kubernetes cluster automatically.

Fixing kubectl Access

After installation, running kubectl get nodes immediately fails:

The connection to the server localhost:8080 was refused

This happens because kubectl defaults to localhost:8080 when no kubeconfig is set. k3s stores its kubeconfig at /etc/rancher/k3s/k3s.yaml. The fix:


mkdir  -p  ~/.kube

sudo  cp  /etc/rancher/k3s/k3s.yaml  ~/.kube/config

sudo  chown  $USER:$USER  ~/.kube/config

Or export it permanently:


echo  'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> ~/.bashrc

source  ~/.bashrc

Retrieve the Node Token

Worker nodes need a token to join the cluster. Grab it from the master:


sudo  cat  /var/lib/rancher/k3s/server/node-token

Keep this value — it is used in every worker join command.

Part 2: Adding the VM Worker (worker1)

Joining the Cluster

On the worker VM, run:


curl  -sfL  https://get.k3s.io | \

K3S_URL=https://192.168.1.44:6443  \

K3S_TOKEN=<node-token> \

sh  -

Problem: Node Password Rejected

The agent started but immediately logged:


Node password rejected, duplicate hostname or contents of

'/etc/rancher/node/password' may not match server node-passwd entry

This happened because the worker VM had previously joined the cluster. k3s stores a node password on both the node (/etc/rancher/node/password) and the master (as a Kubernetes secret). When they don't match, the server rejects the node.

Fix — on the worker:


sudo  systemctl  stop  k3s-agent

sudo  rm  -f  /etc/rancher/node/password

sudo  rm  -rf  /var/lib/rancher/k3s/agent/

sudo  systemctl  start  k3s-agent

Fix — on the master, delete the stale secret:


kubectl  get  secrets  -n  kube-system | grep  node-password

kubectl  delete  secret  worker1.node-password.k3s  -n  kube-system

Problem: Duplicate Hostname

Both the master and worker had the hostname k3s. k3s uses the hostname as the node name, so the server rejected the second node as a duplicate.

Fix — rename the worker:


sudo  hostnamectl  set-hostname  worker1

After renaming and cleaning up the stale secret, the worker joined successfully.

Part 3: The LXC Worker — The Real Story

What is an LXC Container?

LXC (Linux Containers) is a lightweight virtualisation technology. Unlike VMs which emulate full hardware, LXC containers share the host kernel directly. They use Linux namespaces for isolation and cgroups for resource control. They are faster and more efficient than VMs but have less isolation.

Proxmox LXC containers can be privileged (root inside = root on host) or unprivileged (root inside maps to a regular user on host via UID namespacing). Unprivileged is the default and more secure option.

Creating the LXC Container

In Proxmox, I created a Debian Trixie LXC container with:

Joining the Cluster


curl  -sfL  https://get.k3s.io | \

K3S_URL=https://192.168.1.44:6443  \

K3S_TOKEN=<node-token> \

sh  -

The install script ran and printed [INFO] systemd: Starting k3s-agent — and then nothing. It just hung.

Checking the journal:


journalctl  -u  k3s-agent  -f

Error 1: `/dev/kmsg: no such file or directory`


Error: failed to run Kubelet: failed to create kubelet: open /dev/kmsg: no such file or directory

What is /dev/kmsg?

/dev/kmsg is the kernel message buffer device. The Linux kernel uses it to log messages (this is what dmesg reads). kubelet uses it to watch for OOM (Out of Memory) kill events via the oomWatcher. Without it, kubelet refuses to start.

In an unprivileged LXC container, /dev/kmsg does not exist because the container does not have access to kernel devices.

Fix — bind mount from host:

In /etc/pve/lxc/209.conf on the Proxmox host:


lxc.mount.entry: /dev/kmsg dev/kmsg none bind,create=file

This bind mounts the host's /dev/kmsg into the container. Stop and start (not restart) the LXC:


pct  stop  209

pct  start  209

Error 2: `/dev/kmsg: operation not permitted`

After adding the bind mount, the error changed slightly:


open /dev/kmsg: operation not permitted

The file now existed in the container but the process was not allowed to open it. The container was still running in user namespace mode (unprivileged), and AppArmor was blocking the access.

Fix — disable AppArmor restriction:


lxc.apparmor.profile: unconfined

AppArmor is a Linux Security Module that applies mandatory access control policies. The default Proxmox LXC AppArmor profile blocks access to kernel devices like /dev/kmsg. Setting it to unconfined removes all AppArmor restrictions for this container.

Error 3: `/proc/sys/kernel/panic: read-only file system`


Failed to start ContainerManager:

open /proc/sys/kernel/panic: read-only file system

open /proc/sys/kernel/panic_on_oops: read-only file system

open /proc/sys/vm/overcommit_memory: read-only file system

What is /proc/sys?

/proc is a virtual filesystem the kernel exposes so userspace can read and write kernel parameters. /proc/sys/ specifically contains sysctl values — tuneable kernel settings.

kubelet needs to write to these on startup:

kernel/panic — configure kernel panic timeout
kernel/panic_on_oops — whether a kernel oops causes a panic
vm/overcommit_memory — memory overcommit policy

In an unprivileged LXC container, /proc is mounted read-only for safety. Any process inside the container (even root inside) cannot modify these values.

Fix — mount proc and sys as read-write:


lxc.mount.auto: "proc:rw sys:rw"

This tells LXC to mount /proc and /sys with read-write access instead of the default read-only.

Error 4: Various Permission Denied Errors


write /proc/self/oom_score_adj: permission denied

Failed to set sysctl: open /proc/sys/net/netfilter/nf_conntrack_max: permission denied

These were caused by the container still running as unprivileged — the process was root inside the container but mapped to a normal user on the host, so many privileged operations were blocked.

Fix — switch to privileged container:


unprivileged: 0

This is the most significant change. A privileged container maps root inside to actual root on the host. This removes the UID namespace remapping that caused most of the permission errors.

Also needed:


lxc.cgroup2.devices.allow: a

lxc.cap.drop:

cgroup2.devices.allow: a — allows the container access to all devices via the cgroup device controller
cap.drop: (empty) — prevents Proxmox from dropping any Linux capabilities. By default, Proxmox drops capabilities like CAP_SYS_ADMIN, CAP_NET_ADMIN, and CAP_SYS_PTRACE from LXC containers. k3s needs these.

Also needed: `features: keyctl=1,nesting=1`

keyctl=1 — enables the Linux kernel keyring inside the container. containerd uses this to securely store credentials and keys for image pulls.
nesting=1 — enables nested containerisation. k3s runs containerd inside the LXC container, and containerd runs pods (more containers) inside itself. Without nesting enabled, Proxmox blocks the inner container creation.

Final Working LXC Config

After applying all these changes and doing a full pct stop / pct start:


journalctl  -u  k3s-agent  -f

# ... containerd is now running

# ... Server ACTIVE

# ... Started kubelet

Summary: What Each Modification Does

Part 4: LXC as a k3s Worker — Features and Limitations

Features / Advantages

Resource efficiency — LXC containers consume significantly less memory and CPU than VMs. A VM needs a full OS kernel in memory. An LXC container shares the host kernel, so the overhead is minimal. worker2 running k3s uses around 250–300MB RAM idle versus a VM which would use 500MB+ for the OS alone.

Fast startup — LXC containers start in 1–3 seconds versus 15–30 seconds for a VM. For ephemeral worker nodes or autoscaling scenarios this matters.

Storage efficiency — LXC uses the host filesystem directly (with a root filesystem overlay). No separate virtual disk emulation layer. I/O is closer to bare metal performance.

Simple networking — LXC containers participate in the same Proxmox bridge (vmbr0) as VMs. No extra networking configuration is needed for k3s to communicate between the master VM and the LXC worker.

Density — you can run more LXC containers on the same Proxmox host than VMs, making it ideal for testing multi-node cluster topologies on limited hardware.

Limitations

Shared kernel — no kernel version isolation — all LXC containers on a host run the same kernel version as the host. You cannot run a different kernel inside an LXC container. This matters if you need a specific kernel feature or version for your workloads.

Privileged mode is a security trade-off — to get k3s working we had to switch to a privileged container and disable AppArmor. In a privileged container, a root escape inside the container gives root on the host. For a homelab or trusted environment this is acceptable; for production or multi-tenant setups it is a significant risk.

No hardware virtualisation — LXC containers cannot run nested VMs. If your workloads need hardware-level isolation or GPU passthrough in the container, a VM is required.

Kernel module limitations — the LXC container cannot load kernel modules that aren't already loaded on the host. During setup we saw:


modprobe: FATAL: Module br_netfilter not found

These modules need to be loaded on the Proxmox host, not inside the container.

Some syscalls are blocked — even in privileged mode, certain syscalls that could affect the host are restricted. This can cause subtle compatibility issues with some container workloads.

Not suitable for untrusted workloads — because the kernel is shared, a kernel exploit inside an LXC container could theoretically affect the host and all other containers. Never run untrusted code in a privileged LXC container.

Conclusion

Getting k3s running on a Proxmox LXC container is absolutely possible, but it requires understanding why each restriction exists and selectively removing the ones that conflict with k3s's requirements. The journey from a blank LXC to a working cluster node touched on AppArmor, Linux capabilities, cgroups, kernel device access, namespace nesting, and virtual filesystem permissions.

The key takeaway: LXC containers are not VMs. They share the host kernel, and every security restriction that makes them safe is also a potential blocker for complex software like k3s that expects a full OS environment. The solution is not to blindly disable everything — it is to understand each error, trace it to the underlying Linux feature, and make the minimal change required to unblock it.

The final cluster — one control plane VM and two workers (one VM, one LXC) — runs stably with k3s managing scheduling, networking, and DNS across all three nodes via CoreDNS.

I now have a vanilla multi-node Kubernetes cluster running in a Ubuntu VM and an LXC container and accessible from my machine. It’s got nothing deployed inside it yet, but that’s easily fixed.... see u in part 2.

*Built on Proxmox VE with k3s v1.34.6+k3s1 — Debian Trixie LXC — Ubuntu VM nodes

"Why can’t I just mount S3 like a drive?” AWS finally answering that question in 2026

Pendela BhargavaSai — Sun, 12 Apr 2026 13:35:35 +0000

From "why can't I just mount S3 like a drive?" to AWS finally answering that question in 2026.

I've had that conversation more times than I can count.

A developer joins a new AWS project, looks at the architecture, and asks: "We're already storing everything in S3 — why do we also need EFS? Can't we just mount S3 directly?"

And every time, the answer was the same patient explanation about object storage vs file systems, why they're fundamentally different, and why you need separate services for separate workloads. It was the right answer. It just wasn't a satisfying one.

That changed in April 2026 when AWS launched S3 Files — and suddenly that conversation got a lot shorter.

But before we get there, let's start from the beginning. Because understanding why S3 Files matters requires understanding the problem it's solving. And that means understanding the full AWS storage landscape.

The AWS Storage Trinity (Before S3 Files)

AWS has three primary storage services, each built for a completely different purpose. Engineers often get confused because on the surface they all seem to do the same thing: store data. But the way they store it — and who can access it and how — is completely different.

Here's the simplest way I know to think about it:

S3 is like a giant library. You can store billions of books (objects), and anyone with the right access can retrieve any book. But to fix a typo on page 47, you have to reprint the entire book.
EBS is like a hard drive physically attached to your computer. Super fast, but only your computer can use it.
EFS is like a shared office filing cabinet on a network. Anyone in the office can open a drawer, pull out a folder, and edit a document — at the same time.

Let's go deeper on each one.

Amazon S3 — Object Storage Built for Scale

S3 (Simple Storage Service) launched in 2006 and fundamentally changed how the world thinks about storing data. The core idea is simple: you have buckets, and inside buckets you store objects. Each object is just a file plus its metadata, stored at a unique key (think of it like a URL).

What makes S3 special

Virtually unlimited scale. S3 stores more than 500 trillion objects across hundreds of exabytes today.
11 nines of durability (99.999999999%). AWS automatically replicates your data across at least three Availability Zones.
Pay only for what you use. No minimum capacity, no infrastructure to manage.
Multiple storage classes. From S3 Standard (~$0.023/GB) down to Glacier Deep Archive (~$0.00099/GB) for data you almost never touch.

The one thing S3 cannot do

Here's the catch that trips everyone up: S3 is not a file system.

When you store something in S3, it becomes an immutable object. If you want to change even a single character in a file, you have to download the entire object, make your change, and re-upload the whole thing as a new object. There's no such thing as "open this file and edit line 47." That's just not how object storage works.

This isn't a bug — it's by design. The immutability of objects is part of what makes S3 so durable and scalable. But it creates real friction for any workload that needs to work with data the way normal applications do: open a file, read some bytes, write some bytes, save.

# What you can do with S3
aws s3 cp myfile.txt s3://my-bucket/myfile.txt    # upload
aws s3 cp s3://my-bucket/myfile.txt ./myfile.txt  # download
aws s3 rm s3://my-bucket/myfile.txt               # delete

# What you CANNOT do
# Open myfile.txt and append a line — impossible without full re-upload

Amazon EBS — The Fast Attached Drive

EBS (Elastic Block Store) is block storage — the AWS equivalent of an SSD attached directly to your server. When you launch an EC2 instance, the root volume (where the operating system lives) is an EBS volume.

What EBS is good at

Speed. EBS delivers single-digit millisecond latency because it behaves like a local disk.
POSIX semantics. You can open files, write individual bytes, seek to specific positions — everything a normal file system supports.
Consistency. What you write is immediately readable. No eventual consistency concerns.

The hard limit of EBS

EBS volumes can only be attached to one EC2 instance at a time (with some multi-attach exceptions for specific use cases).

This means if you have a cluster of 10 EC2 instances all running your application, each one needs its own EBS volume. They can't share data through EBS. If instance A writes a file, instance B can't see it without some kind of sync mechanism.

EC2 Instance A  →  EBS Volume A  (can't share)
EC2 Instance B  →  EBS Volume B  (separate, isolated)
EC2 Instance C  →  EBS Volume C  (separate, isolated)

For single-instance workloads — databases, operating system volumes, single-server applications — EBS is excellent. The moment you need shared storage across multiple servers, you hit a wall.

Amazon EFS — The Shared Network Drive

EFS (Elastic File System) is AWS's managed Network File System (NFS). Think of it as a shared drive that any number of EC2 instances, containers, or Lambda functions can mount simultaneously and use like a local file system.

What EFS solves

Concurrent access. Thousands of compute resources can mount and use the same EFS volume at the same time.
Full POSIX semantics. Open files, edit bytes in-place, file locking, directory operations — everything works.
Scales automatically. The file system grows and shrinks as you add or remove files. No capacity planning required.
Sub-millisecond latency on Standard tier.

EC2 Instance A  ──┐
EC2 Instance B  ──┤──→  EFS Volume  (all share the same files)
EC2 Instance C  ──┘
Lambda Function ──┘

Where EFS falls short

The pricing model. EFS charges you for every gigabyte stored, whether you touched it this month or not. Standard tier is $0.30/GB-month — roughly 13x more expensive than S3 Standard per gigabyte.

This is fine when your data is "hot" (actively accessed). It's painful when you have petabytes of data where only a fraction is actively used at any time. You end up paying full file system prices for data that's sitting idle.

And the other problem: EFS has zero native integration with S3. They're completely separate systems. Your data lake is in S3. Your compute needs EFS. So you write sync scripts to copy data back and forth — and now you have two copies of everything, two storage bills, and a manual process that breaks at the worst possible times.

The Old Workflow Pain (The Problem All of This Creates)

Before S3 Files, a typical ML or data engineering team's workflow looked like this:

S3 Data Lake
    ↓  (manual copy — takes time, costs money)
EFS Volume
    ↓  (mount on EC2)
EC2 Training Job
    ↓  (output back to EFS)
    ↓  (another manual copy)
S3 Data Lake  ← results stored here for analytics

Every arrow in that diagram is a point of failure. Every copy step is a delay, a cost, and a potential for the two copies to drift out of sync. Engineers were spending real engineering hours maintaining these sync pipelines — hours that weren't building anything valuable.

This is the problem that s3fs tried to solve, years before AWS had an official answer.

s3fs-fuse — The Community's Workaround

If you've been working with AWS for a few years, you've probably encountered s3fs-fuse. It's an open-source FUSE (Filesystem in Userspace) tool that lets you mount an S3 bucket as a local directory on Linux, macOS, or FreeBSD.

# Install
sudo apt-get install s3fs

# Configure credentials
echo "ACCESS_KEY_ID:SECRET_ACCESS_KEY" > ~/.passwd-s3fs
chmod 600 ~/.passwd-s3fs

# Mount your bucket
s3fs my-bucket /mnt/s3-data -o passwd_file=~/.passwd-s3fs

After that, you can run ls, cp, cat — your S3 bucket looks like a local folder. For a quick demo or a simple use case, it feels magical.

What's actually happening under the hood

Here's the thing nobody tells you upfront: s3fs isn't really giving you file system access to S3. It's translating file commands into S3 API calls — and the translation has serious limitations.

When you "edit" a file through s3fs, this is what actually happens:

You: nano myfile.txt  (make a small change, save)
     ↓
s3fs: GET entire object from S3 → download to local temp cache
s3fs: You edit the local temp copy
s3fs: On file close → PUT entire object back to S3 (full re-upload)

Change one character in a 10GB file? s3fs downloads all 10GB, makes the change, and uploads all 10GB again. Every time.

The real limitations you need to know

No file locking. If two processes try to write to the same file through s3fs at the same time, you get data corruption. Not an error message — silent data corruption.

No atomic renames. Renaming a file in s3fs copies it to a new key and deletes the old one. Any application that relies on atomic renames (which includes most databases and many log processors) will break.

Slow directory listings. Every ls is a ListObjects API call to S3. On a bucket with millions of objects, this is painfully slow.

No hard links or symbolic links. S3 simply doesn't support them.

Operation          | What s3fs does              | Problem
-------------------|-----------------------------|-----------------------
Read file          | GET entire object           | Slow for large files
Edit file          | Download → edit → full PUT  | Expensive re-upload
Append to file     | Rewrite entire object       | Very expensive
Rename file        | Copy + Delete               | Not atomic
File lock          | Not supported               | Data corruption risk
List directory     | ListObjects API call        | Slow on large buckets

s3fs works well for lightweight, read-heavy, single-process use cases. But the moment you need multi-process access, in-place edits, or production reliability — it starts breaking down. The community built it because AWS didn't have a better answer. Eventually, AWS tried building their own version.

Mountpoint for S3 — AWS's Open-Source Attempt (2023)

In 2023, AWS released Mountpoint for S3, their own open-source FUSE client. It was faster than s3fs-fuse and better optimised for cloud-native read-heavy workloads.

But it still couldn't do in-place edits, directory renames, or file locking. It was better than s3fs-fuse, but it still hit the same fundamental ceiling: you can't make S3's API behave like a real file system by pretending.

AWS knew this. Internally, they'd been trying to solve it properly for years.

Amazon S3 Files — The Real Solution (April 2026)

On April 7, 2026, AWS launched S3 Files — and it's the most significant S3 update since the service launched.

The internal project was even called "EFS3" at one point. One engineer on the team described the design process as "a battle of unpalatable compromises." Getting object storage and file system semantics to truly coexist is genuinely hard engineering. Every design decision forced a tradeoff where either the file presentation or the object presentation had to give something up.

What they landed on is clever: instead of trying to make the S3 API behave like a file system (which is what s3fs does), they did the opposite — they took a real, production-grade file system (EFS) and connected it directly to S3 storage.

How S3 Files actually works

S3 Files uses a two-tier architecture:

Tier 1 — EFS Cache Layer (hot data)

Stores your active working set: recently written files, recently read files, metadata
Delivers ~1ms latency
Serves small files (under 128KB by default) entirely from cache
Handles all NFS file operations — open, read, write, rename, lock

Tier 2 — S3 Bucket (your full dataset)

Holds your complete data at normal S3 prices (~$0.023/GB)
Large reads (1MB+) bypass the cache entirely and stream directly from S3 for free
Changes made through the file system sync back to S3 automatically within minutes

Your Application
      ↓  (NFS mount — standard Linux file operations)
EFS Cache Layer  ←→  Smart Router
      ↓                    ↓
   Hot data            Cold/large data
   (~1ms)              (streams from S3, free)
      ↓                    ↓
      └────────────────────┘
                  ↓
            S3 Bucket
       (your data, always here)

The key insight: your data never leaves S3. The EFS cache is just a smart caching layer on top. You're not maintaining two copies — you have one copy in S3, accessible via both the S3 API and the file system mount simultaneously.

OLD way to New way

Getting started in 3 steps

Step 1: Create an S3 file system

In the AWS Console → S3 → File Systems → Create file system. Enter your bucket name, done.

Or via CLI:

aws s3api create-file-system --bucket my-bucket
aws s3api create-mount-target --file-system-id fs-xxxx --subnet-id subnet-xxxx

Step 2: Mount it on your EC2 instance

Make sure the amazon-efs-utils package is installed (preinstalled on AWS AMIs), then:

sudo mkdir /mnt/s3files
sudo mount -t s3files fs-0aa860d05df9afdfe:/ /mnt/s3files

Step 3: Use it like any local directory

# Create a file
echo "Hello S3 Files" > /mnt/s3files/hello.txt

# Edit it in place
echo "New line added" >> /mnt/s3files/hello.txt

# List files
ls -la /mnt/s3files/

# The same data is accessible via S3 API too
aws s3 ls s3://my-bucket/

Changes you make through the file system mount appear in S3 within minutes. Changes made directly to the S3 bucket appear in the file system within seconds.

Security — what you need to know

IAM integration for access control at both file system and object level
Data encrypted in transit using TLS 1.3
Data encrypted at rest using SSE-S3 (or KMS if you prefer customer-managed keys)
POSIX permissions (UID/GID) stored as S3 object metadata
Monitor via CloudWatch metrics and CloudTrail logs

Pricing — the part that actually makes sense

S3 Files charges EFS-level rates, but only on the fraction of data you're actively working with:

What you pay for	Rate
High-performance storage (hot data)	$0.30/GB-month
Reads (small files served from cache)	$0.03/GB
Writes	$0.06/GB
Everything else in your S3 bucket	Standard S3 rates (~$0.023/GB)

If you have a 100TB dataset but only 1TB is actively used at any time — you pay EFS rates on 1TB and S3 rates on the other 99TB. AWS claims up to 90% cost savings compared to the old pattern of cycling data between S3 and a dedicated EFS volume.

Putting It All Together — Which Service Should You Use?

Here's the honest answer:

Use this	When you need
S3	Bulk storage, backups, data lakes, analytics, static assets, anything accessed via API
EBS	OS volumes, databases, single-instance high-performance storage
EFS	Shared file system for legacy NAS migration, on-premises workloads moving to cloud, apps that need pure NFS without S3
S3 Files	ML pipelines, agentic AI workflows, data engineering, any workload where both S3 API and file system access are needed
s3fs-fuse	Quick prototypes, read-heavy single-process scripts, legacy apps where you can't change the architecture

The quick comparison

Why This Matters for ML and AI Workloads

If you're building machine learning pipelines or agentic AI systems, S3 Files is worth paying close attention to.

The old workflow was: data lives in S3 → copy to EFS before training → run training job → copy results back to S3. For large datasets, that copy step alone could take hours. You were also paying double storage costs during the transition.

With S3 Files, your training job mounts the S3 bucket directly. The EFS cache warms up as your training reads data. No copy step. No sync script. No duplicate storage.

For agentic AI systems specifically — where multiple agents need to coordinate through shared files, read from each other's outputs, maintain shared state — S3 Files provides exactly the concurrent NFS access with close-to-open consistency that these workloads need. Standard Python file operations, standard shell tools, all working against data that lives in S3.

The Short Version

For a decade, AWS storage was a choice: pay S3 prices and lose file system semantics, or pay EFS prices and lose S3 integration. Teams wrote sync scripts, maintained duplicate data, and spent engineering time on storage plumbing instead of actual product work.

s3fs-fuse was the community's best attempt at a workaround — and it worked, up to a point. But it was always emulating file system behavior on top of an API that wasn't designed for it.

S3 Files is the first time AWS has genuinely solved this at the right layer. Real NFS semantics, real S3 storage, real production reliability. One bucket, two protocols, no compromises.

If you've ever maintained a sync script between your data lake and your compute layer — you know exactly what problem this solves. And you know exactly how good it feels to delete that script.

Resources

Published April 2026. All pricing figures reflect us-east-1 as of the time of writing.

If this helped you, drop a reaction or leave a comment — curious what storage patterns others are running into in the wild.

DEV Community: Pendela BhargavaSai

K3s vs Kubernetes: A Deep Dive into Control Plane Architecture

🧠 Why This Post Exists

🏗️ The Kubernetes Control Plane: A Ground-Up Look

1. 🔵 kube-apiserver — The Brain's Frontal Lobe

What It Actually Does

The Watch Mechanism — The Heartbeat of Kubernetes

Aggregation Layer & CRDs

Production Tuning Knobs

2. 🟣 etcd — The Distributed Brain's Memory

What etcd Actually Is

Raft in Plain English

How Kubernetes Data Lives in etcd

MVCC — Multi-Version Concurrency Control

etcd Failure Modes You Must Know

3. 🟡 kube-scheduler — The CPU-Time Auctioneer

What It Actually Does

The Scheduling Framework — Two-Phase Deep Dive

Preemption — What Happens When No Node Passes Filtering

The Binding Cache — Optimistic Concurrency

4. 🟢 kube-controller-manager — The Reconciliation Engine

What It Actually Is

Key Controllers and What They Actually Do

Informer + WorkQueue Architecture

5. 🔴 cloud-controller-manager — The Cloud API Bridge

What It Actually Does

⚡ The K3s Control Plane: Architectural Reimagination

K3s Single Binary Philosophy

1. K3s API Server — Same Core, Slimmer Defaults

2. SQLite / Kine — The etcd Abstraction Layer

3. K3s Controller Manager — Pruned and Extended

4. K3s Scheduler — Unchanged but Co-located

5. The Flannel CNI — Embedded Networking

📊 Side-by-Side Deep Comparison

🔑 When to Choose What

Choose Standard Kubernetes When:

Choose K3s When:

🔭 The Architecture Decision Tree

🎯 Closing Thoughts

I Broke My Proxmox Home Lab with a GPU Passthrough - Here's How I Fixed It

What I Was Trying to Do

What Happened — The Crash Loop

Understanding the Problem — Why It Was So Hard to Escape

The Solution — Masking the VM Start Service at Boot

Step 1: Stop the autoboot timer at GRUB

Step 2: Edit the boot command

What systemd.mask=pve-guests.service actually does

Step 3: Fix the VM config via web UI

Plot Twist — The GPU Renamed Itself

Root Cause Analysis

Lessons Learned

The Final State

Quick Reference — Commands Used

The Ultimate Guide to Kubernetes Load Balancers in 2026 (K3s Edition)

🧭 Why This Guide Exists

🗺️ The Landscape: What Are We Even Comparing?

🔬 Competitor Deep-Dive

1. 🏠 Klipper ServiceLB (K3s Built-In)

2. 🟢 NGINX Ingress Controller

3. 🐹 Traefik (K3s Default)

4. 🔷 MetalLB

5. ⚡ HAProxy Ingress Controller

6. 🌊 Envoy Proxy

7. 🕸️ Istio (Service Mesh)

8. 🔗 Linkerd (Service Mesh)

9. 🧬 Cilium (eBPF-based CNI + Service Mesh)

📊 The Big Comparison Table

🏗️ Architecture Patterns for K3s

Pattern 1: Minimal (Single Node / Homelab)

Pattern 2: Bare-Metal Production (Most Common)

Pattern 3: High-Performance Production

Pattern 4: Secure Microservices (Security-First)

Pattern 5: Maximum Performance + Security (Advanced)

🏎️ Performance Benchmarks at a Glance

🎯 Decision Framework: Which One for Your K3s Cluster?

🔧 K3s-Specific Tips & Gotchas

🏁 Final Recommendations

📚 Further Reading

Kubernetes CNI Complete Guide: Flannel vs Cilium vs Calico + Cloud Provider CNIs

Table of Contents

What `systemd.mask=pve-guests.service` actually does

The lesson about `pull_request_target`