Recently, I’ve been experimenting with different networking configurations for a Talos Linux Kubernetes cluster deployed in hybrid mode - with control plane nodes running in AWS and a worker node hosted on-premises in QEMU. My goal was to evaluate how Cilium CNI behaves in such a setup, especially when combined with KubeSpan, Talos’ native WireGuard-based mesh networking layer.
In this post, I’ll share my findings from three different setups, highlighting the challenges, performance results, and takeaways for hybrid environments.
1. Cilium Native WireGuard with KubeSpan Disabled
My first experiment was running Cilium’s native WireGuard encryption while disabling KubeSpan.
On paper, this should provide secure pod-to-pod communication. In practice, it failed. The reason lies in how Cilium implements WireGuard - it assumes direct IP connectivity between nodes.
In my hybrid setup, the on-prem worker lives behind NAT, which makes it unreachable for AWS nodes. Since Cilium does not support NAT traversal techniques (e.g., hole punching or STUN-like mechanisms), the WireGuard handshake could not be established.
This is exactly where KubeSpan shines. Unlike Cilium’s implementation, KubeSpan was designed for hybrid, cloud, and NAT-constrained topologies. It automatically builds WireGuard tunnels across boundaries, enabling connectivity even when nodes are hidden behind NAT.
Takeaway: Without KubeSpan, Cilium WireGuard isn’t viable in hybrid deployments with NAT.
2. Native Routing to Reduce VXLAN Encapsulation
The second setup explored Cilium’s native routing as a way to reduce VXLAN encapsulation overhead. VXLAN is fine in co-located clusters, but it adds overhead, especially in cross-node, hybrid traffic.
At first, I assumed native routing wouldn’t work outside of tightly connected environments. However, with a few tweaks it became possible:
Deploy a DaemonSet that extracts each node’s cilium_host IP.
Assign a secondary IP with a wider subnet mask (e.g., /24).
Enable advertiseKubernetesNetworks so that pod CIDRs are shared across nodes.
Ensure KubeSpan peers include these CIDRs in their AllowedIPs.
This workaround allowed Cilium to operate in native routing mode, bypassing VXLAN encapsulation even in the hybrid cluster.
Takeaway: With some custom plumbing, native routing works across NAT-boundaries when combined with KubeSpan.
3. Test Results
I ran a series of TCP/UDP performance benchmarks. The full dataset is available here
but here’s the summary:
Stream Tests: These tests were instrumental in evaluating the throughput performance of pods and nodes.
RR tests (Request-Response): These tests allowed us to assess the packet per second and latency performance of pods and nodes.
CRR tests (Connect-Request-Response): By utilizing this scenario, we could evaluate the New Connection Per Second performance of pods and nodes.
----------------------------------------------------------------------------------------
KubeSpan Enabled. Cilium settings:
k8sServiceHost: localhost
k8sServicePort: 7445
kubeProxyReplacement: true
enableK8sEndpointSlice: true
localRedirectPolicy: true
healthChecking: true
bpf:
masquerade: true
ipv4:
enabled: true
hostServices:
enabled: true
hostPort:
enabled: true
nodePort:
enabled: true
externalIPs:
enabled: true
hostFirewall:
enabled: true
----------------------------------------------------------------------------------------
cilium connectivity perf --tolerations "" --namespace-labels pod-security.kubernetes.io/enforce=privileged -n kube-system --helm-release-name cilium --udp --crr --samples 3 \
--node-selector-client "kubernetes.io/hostname=io-apps-bootstrap-1" --node-selector-server "kubernetes.io/hostname=io-gpu-pruuzglzan18m9y8"
🔥 Network Performance Test Summary - NON COLOCATED NODES (AWS->ONPREM):
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 Scenario | Node | Test | Duration | Min | Mean | Max | P50 | P90 | P99 | Transaction rate OP/s
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 pod-to-pod | same-node | TCP_CRR | 10s | 57µs | 67.26µs | 271µs | 65µs | 73µs | 101µs | 14822.78
📋 pod-to-pod | same-node | TCP_RR | 10s | 23µs | 33.84µs | 201µs | 33µs | 36µs | 49µs | 29445.81
📋 pod-to-pod | same-node | UDP_RR | 10s | 21µs | 33.42µs | 19.48ms | 33µs | 37µs | 50µs | 29795.95
📋 host-to-host | same-node | TCP_CRR | 10s | 67µs | 87.43µs | 349µs | 86µs | 99µs | 128µs | 11405.75
📋 host-to-host | same-node | TCP_RR | 10s | 25µs | 36.54µs | 227µs | 36µs | 39µs | 52µs | 27270.72
📋 host-to-host | same-node | UDP_RR | 10s | 21µs | 30.35µs | 215µs | 30µs | 31µs | 46µs | 32813.49
📋 pod-to-pod | other-node | TCP_CRR | 10s | 287.284ms | 288.18447ms | 294.926ms | 285.312ms | 289.375ms | 295ms | 3.37
📋 pod-to-pod | other-node | TCP_RR | 10s | 143.513ms | 146.13599ms | 287.71ms | 145.074ms | 149.104ms | 150ms | 6.75
📋 pod-to-pod | other-node | UDP_RR | 10s | 143.883ms | 144.18403ms | 144.613ms | 144.927ms | 148.985ms | 149.855ms | 6.90
📋 host-to-host | other-node | TCP_CRR | 10s | 287.289ms | 287.86662ms | 289.423ms | 285ms | 288.823ms | 289.705ms | 3.38
📋 host-to-host | other-node | TCP_RR | 10s | 143.572ms | 146.06574ms | 288.202ms | 145.074ms | 149.104ms | 150ms | 6.75
📋 host-to-host | other-node | UDP_RR | 10s | 143.486ms | 144.03764ms | 146.835ms | 144.927ms | 148.985ms | 149.855ms | 6.90
📋 pod-to-pod | same-node | TCP_CRR | 10s | 54µs | 66.73µs | 306µs | 65µs | 73µs | 97µs | 14935.31
📋 pod-to-pod | same-node | TCP_RR | 10s | 23µs | 33.89µs | 177µs | 33µs | 36µs | 50µs | 29391.89
📋 pod-to-pod | same-node | UDP_RR | 10s | 21µs | 33.05µs | 195µs | 32µs | 35µs | 49µs | 30139.94
📋 host-to-host | same-node | TCP_CRR | 10s | 68µs | 85.73µs | 366µs | 86µs | 92µs | 118µs | 11630.99
📋 host-to-host | same-node | TCP_RR | 10s | 24µs | 36.82µs | 5.879ms | 36µs | 39µs | 53µs | 27060.31
📋 host-to-host | same-node | UDP_RR | 10s | 22µs | 31.34µs | 21.676ms | 30µs | 38µs | 48µs | 31787.03
📋 pod-to-pod | other-node | TCP_CRR | 10s | 287.141ms | 287.72271ms | 288.986ms | 285ms | 288.823ms | 289.705ms | 3.38
📋 pod-to-pod | other-node | TCP_RR | 10s | 143.716ms | 146.31566ms | 287.914ms | 145.074ms | 149.104ms | 150ms | 6.74
📋 pod-to-pod | other-node | UDP_RR | 10s | 143.48ms | 144.11599ms | 149.036ms | 144.927ms | 148.985ms | 149.855ms | 6.90
📋 host-to-host | other-node | TCP_CRR | 10s | 287.004ms | 287.91197ms | 292.78ms | 285.312ms | 289.375ms | 295ms | 3.37
📋 host-to-host | other-node | TCP_RR | 10s | 145.089ms | 147.76345ms | 290.773ms | 145ms | 149.09ms | 150ms | 6.67
📋 host-to-host | other-node | UDP_RR | 10s | 145.184ms | 145.58032ms | 151.39ms | 145.074ms | 149.104ms | 150ms | 6.80
📋 pod-to-pod | same-node | TCP_CRR | 10s | 56µs | 68.28µs | 284µs | 65µs | 82µs | 106µs | 14600.08
📋 pod-to-pod | same-node | TCP_RR | 10s | 22µs | 34.21µs | 236µs | 33µs | 37µs | 50µs | 29128.28
📋 pod-to-pod | same-node | UDP_RR | 10s | 20µs | 32.74µs | 209µs | 32µs | 35µs | 48µs | 30413.30
📋 host-to-host | same-node | TCP_CRR | 10s | 67µs | 85.69µs | 367µs | 85µs | 92µs | 119µs | 11638.88
📋 host-to-host | same-node | TCP_RR | 10s | 23µs | 36.68µs | 208µs | 36µs | 39µs | 53µs | 27172.15
📋 host-to-host | same-node | UDP_RR | 10s | 20µs | 30.66µs | 3.662ms | 30µs | 32µs | 46µs | 32483.28
📋 pod-to-pod | other-node | TCP_CRR | 10s | 290.464ms | 291.16426ms | 292.808ms | 295ms | 298.823ms | 299.705ms | 3.40
📋 pod-to-pod | other-node | TCP_RR | 10s | 145.063ms | 148.08606ms | 295.874ms | 145ms | 149.09ms | 150ms | 6.66
📋 pod-to-pod | other-node | UDP_RR | 10s | 144.919ms | 145.82088ms | 149.155ms | 145ms | 148.97ms | 149.852ms | 6.80
📋 host-to-host | other-node | TCP_CRR | 10s | 287.323ms | 287.86053ms | 289.85ms | 285ms | 288.823ms | 289.705ms | 3.37
📋 host-to-host | other-node | TCP_RR | 10s | 143.508ms | 146.03766ms | 288.42ms | 145.074ms | 149.104ms | 150ms | 6.75
📋 host-to-host | other-node | UDP_RR | 10s | 143.523ms | 143.91538ms | 145.713ms | 144.927ms | 148.985ms | 149.855ms | 6.90
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
📋 Scenario | Node | Test | Duration | Throughput Mb/s
----------------------------------------------------------------------------------------
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 12950.39
📋 pod-to-pod | same-node | UDP_STREAM | 10s | 2092.67
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 48431.66
📋 pod-to-pod | same-node | UDP_STREAM_MULTI | 10s | 7942.53
📋 host-to-host | same-node | TCP_STREAM | 10s | 20550.47
📋 host-to-host | same-node | UDP_STREAM | 10s | 1442.80
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 79811.91
📋 host-to-host | same-node | UDP_STREAM_MULTI | 10s | 5397.83
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 105.28
📋 pod-to-pod | other-node | UDP_STREAM | 10s | 351.63
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 380.95
📋 pod-to-pod | other-node | UDP_STREAM_MULTI | 10s | 496.31
📋 host-to-host | other-node | TCP_STREAM | 10s | 104.16
📋 host-to-host | other-node | UDP_STREAM | 10s | 399.63
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 430.81
📋 host-to-host | other-node | UDP_STREAM_MULTI | 10s | 536.22
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 12556.70
📋 pod-to-pod | same-node | UDP_STREAM | 10s | 2071.79
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 48310.76
📋 pod-to-pod | same-node | UDP_STREAM_MULTI | 10s | 7941.07
📋 host-to-host | same-node | TCP_STREAM | 10s | 20379.97
📋 host-to-host | same-node | UDP_STREAM | 10s | 1410.00
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 79800.63
📋 host-to-host | same-node | UDP_STREAM_MULTI | 10s | 5430.72
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 103.53
📋 pod-to-pod | other-node | UDP_STREAM | 10s | 325.66
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 413.59
📋 pod-to-pod | other-node | UDP_STREAM_MULTI | 10s | 480.75
📋 host-to-host | other-node | TCP_STREAM | 10s | 119.24
📋 host-to-host | other-node | UDP_STREAM | 10s | 413.51
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 383.96
📋 host-to-host | other-node | UDP_STREAM_MULTI | 10s | 437.77
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 12784.47
📋 pod-to-pod | same-node | UDP_STREAM | 10s | 2079.99
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 49080.75
📋 pod-to-pod | same-node | UDP_STREAM_MULTI | 10s | 7867.70
📋 host-to-host | same-node | TCP_STREAM | 10s | 20745.23
📋 host-to-host | same-node | UDP_STREAM | 10s | 1432.03
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 79912.94
📋 host-to-host | same-node | UDP_STREAM_MULTI | 10s | 5489.68
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 101.82
📋 pod-to-pod | other-node | UDP_STREAM | 10s | 329.69
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 407.10
📋 pod-to-pod | other-node | UDP_STREAM_MULTI | 10s | 470.26
📋 host-to-host | other-node | TCP_STREAM | 10s | 107.29
📋 host-to-host | other-node | UDP_STREAM | 10s | 404.08
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 405.85
📋 host-to-host | other-node | UDP_STREAM_MULTI | 10s | 567.32
----------------------------------------------------------------------------------------
cilium connectivity perf --tolerations "" --namespace-labels pod-security.kubernetes.io/enforce=privileged -n kube-system --helm-release-name cilium --udp --crr --samples 3 \
--node-selector-client "kubernetes.io/hostname=io-apps-bootstrap-1" --node-selector-server "kubernetes.io/hostname=io-controlplane-1"
🔥 Network Performance Test Summary - COLOCATED NODES (AWS->AWS):
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 Scenario | Node | Test | Duration | Min | Mean | Max | P50 | P90 | P99 | Transaction rate OP/s
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 pod-to-pod | same-node | TCP_CRR | 10s | 93µs | 153.92µs | 11.063ms | 136µs | 203µs | 442µs | 6473.15
📋 pod-to-pod | same-node | TCP_RR | 10s | 31µs | 50.49µs | 29.252ms | 46µs | 62µs | 121µs | 19644.46
📋 pod-to-pod | same-node | UDP_RR | 10s | 30µs | 49.49µs | 6.326ms | 48µs | 63µs | 123µs | 20042.63
📋 host-to-host | same-node | TCP_CRR | 10s | 99µs | 157.13µs | 11.857ms | 139µs | 203µs | 437µs | 6340.76
📋 host-to-host | same-node | TCP_RR | 10s | 34µs | 51.88µs | 6.772ms | 49µs | 65µs | 124µs | 19121.01
📋 host-to-host | same-node | UDP_RR | 10s | 33µs | 51.49µs | 6.29ms | 47µs | 64µs | 128µs | 19250.97
📋 pod-to-pod | other-node | TCP_CRR | 10s | 805µs | 1.42404ms | 1.031978s | 1.064ms | 1.333ms | 2.533ms | 701.76
📋 pod-to-pod | other-node | TCP_RR | 10s | 305µs | 509.3µs | 17.769ms | 449µs | 608µs | 1.67ms | 1961.27
📋 pod-to-pod | other-node | UDP_RR | 10s | 296µs | 486.33µs | 25.148ms | 433µs | 565µs | 1.495ms | 2053.80
📋 host-to-host | other-node | TCP_CRR | 10s | 717µs | 1.11209ms | 19.207ms | 997µs | 1.348ms | 3.325ms | 898.50
📋 host-to-host | other-node | TCP_RR | 10s | 273µs | 441.12µs | 10.452ms | 406µs | 514µs | 1.147ms | 2264.47
📋 host-to-host | other-node | UDP_RR | 10s | 270µs | 427.96µs | 16.015ms | 394µs | 504µs | 1.031ms | 2333.99
📋 pod-to-pod | same-node | TCP_CRR | 10s | 94µs | 150.7µs | 11.579ms | 134µs | 198µs | 417µs | 6610.99
📋 pod-to-pod | same-node | TCP_RR | 10s | 31µs | 48.94µs | 13.847ms | 46µs | 61µs | 123µs | 20268.63
📋 pod-to-pod | same-node | UDP_RR | 10s | 30µs | 49.32µs | 16.98ms | 47µs | 63µs | 139µs | 20120.57
📋 host-to-host | same-node | TCP_CRR | 10s | 101µs | 158.66µs | 20.761ms | 143µs | 205µs | 438µs | 6280.68
📋 host-to-host | same-node | TCP_RR | 10s | 31µs | 52.56µs | 32.051ms | 49µs | 65µs | 124µs | 18856.14
📋 host-to-host | same-node | UDP_RR | 10s | 30µs | 52.06µs | 18.874ms | 47µs | 63µs | 132µs | 19055.07
📋 pod-to-pod | other-node | TCP_CRR | 10s | 778µs | 1.14343ms | 14.77ms | 1.065ms | 1.385ms | 2.584ms | 873.62
📋 pod-to-pod | other-node | TCP_RR | 10s | 300µs | 473.68µs | 18.28ms | 439µs | 559µs | 1.154ms | 2108.58
📋 pod-to-pod | other-node | UDP_RR | 10s | 305µs | 468.16µs | 12.304ms | 431µs | 550µs | 1.155ms | 2133.48
📋 host-to-host | other-node | TCP_CRR | 10s | 679µs | 1.01739ms | 20.003ms | 948µs | 1.203ms | 2.3ms | 982.19
📋 host-to-host | other-node | TCP_RR | 10s | 290µs | 451.32µs | 8.471ms | 400µs | 529µs | 1.5ms | 2213.05
📋 host-to-host | other-node | UDP_RR | 10s | 275µs | 438.89µs | 48.143ms | 392µs | 512µs | 1.178ms | 2275.78
📋 pod-to-pod | same-node | TCP_CRR | 10s | 91µs | 154.91µs | 9.155ms | 136µs | 206µs | 463µs | 6433.15
📋 pod-to-pod | same-node | TCP_RR | 10s | 31µs | 49.53µs | 11.302ms | 46µs | 62µs | 119µs | 20018.91
📋 pod-to-pod | same-node | UDP_RR | 10s | 30µs | 50.66µs | 6.199ms | 48µs | 64µs | 127µs | 19585.08
📋 host-to-host | same-node | TCP_CRR | 10s | 97µs | 161.53µs | 19.182ms | 140µs | 199µs | 432µs | 6169.06
📋 host-to-host | same-node | TCP_RR | 10s | 33µs | 51.89µs | 9.37ms | 49µs | 64µs | 120µs | 19116.27
📋 host-to-host | same-node | UDP_RR | 10s | 33µs | 51.96µs | 22.041ms | 47µs | 65µs | 133µs | 19078.59
📋 pod-to-pod | other-node | TCP_CRR | 10s | 785µs | 1.17809ms | 14.512ms | 1.088ms | 1.45ms | 2.93ms | 848.22
📋 pod-to-pod | other-node | TCP_RR | 10s | 290µs | 491.34µs | 23.311ms | 445µs | 574µs | 1.358ms | 2032.95
📋 pod-to-pod | other-node | UDP_RR | 10s | 316µs | 530.1µs | 41.495ms | 455µs | 606µs | 1.642ms | 1884.38
📋 host-to-host | other-node | TCP_CRR | 10s | 714µs | 1.06433ms | 31.965ms | 963µs | 1.266ms | 2.766ms | 938.79
📋 host-to-host | other-node | TCP_RR | 10s | 279µs | 443.92µs | 18.193ms | 402µs | 516µs | 1.246ms | 2249.97
📋 host-to-host | other-node | UDP_RR | 10s | 268µs | 420.99µs | 24.753ms | 389µs | 495µs | 1.05ms | 2372.48
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
📋 Scenario | Node | Test | Duration | Throughput Mb/s
----------------------------------------------------------------------------------------
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 7450.22
📋 pod-to-pod | same-node | UDP_STREAM | 10s | 845.16
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 11222.14
📋 pod-to-pod | same-node | UDP_STREAM_MULTI | 10s | 849.59
📋 host-to-host | same-node | TCP_STREAM | 10s | 18252.45
📋 host-to-host | same-node | UDP_STREAM | 10s | 775.24
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 23872.71
📋 host-to-host | same-node | UDP_STREAM_MULTI | 10s | 871.62
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 1474.15
📋 pod-to-pod | other-node | UDP_STREAM | 10s | 322.32
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 1547.00
📋 pod-to-pod | other-node | UDP_STREAM_MULTI | 10s | 472.29
📋 host-to-host | other-node | TCP_STREAM | 10s | 1844.26
📋 host-to-host | other-node | UDP_STREAM | 10s | 394.33
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 1816.71
📋 host-to-host | other-node | UDP_STREAM_MULTI | 10s | 546.08
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 8057.11
📋 pod-to-pod | same-node | UDP_STREAM | 10s | 717.48
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 11106.95
📋 pod-to-pod | same-node | UDP_STREAM_MULTI | 10s | 754.54
📋 host-to-host | same-node | TCP_STREAM | 10s | 17977.78
📋 host-to-host | same-node | UDP_STREAM | 10s | 756.63
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 24229.15
📋 host-to-host | same-node | UDP_STREAM_MULTI | 10s | 843.12
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 1641.48
📋 pod-to-pod | other-node | UDP_STREAM | 10s | 349.99
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 1595.12
📋 pod-to-pod | other-node | UDP_STREAM_MULTI | 10s | 463.93
📋 host-to-host | other-node | TCP_STREAM | 10s | 1769.24
📋 host-to-host | other-node | UDP_STREAM | 10s | 410.02
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 1833.89
📋 host-to-host | other-node | UDP_STREAM_MULTI | 10s | 574.86
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 7852.78
📋 pod-to-pod | same-node | UDP_STREAM | 10s | 739.25
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 11195.04
📋 pod-to-pod | same-node | UDP_STREAM_MULTI | 10s | 870.62
📋 host-to-host | same-node | TCP_STREAM | 10s | 18268.06
📋 host-to-host | same-node | UDP_STREAM | 10s | 783.99
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 23901.10
📋 host-to-host | same-node | UDP_STREAM_MULTI | 10s | 751.55
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 1507.56
📋 pod-to-pod | other-node | UDP_STREAM | 10s | 328.49
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 1498.47
📋 pod-to-pod | other-node | UDP_STREAM_MULTI | 10s | 442.88
📋 host-to-host | other-node | TCP_STREAM | 10s | 1746.56
📋 host-to-host | other-node | UDP_STREAM | 10s | 431.39
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 1821.09
📋 host-to-host | other-node | UDP_STREAM_MULTI | 10s | 574.39
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
KubeSpan Enabled. Native Routing enabled.
k8sServiceHost: localhost
k8sServicePort: 7445
kubeProxyReplacement: true
enableK8sEndpointSlice: true
localRedirectPolicy: true
healthChecking: true
routingMode: native
ipv4NativeRoutingCIDR: "10.244.0.0/16"
bpf:
masquerade: true
hostLegacyRouting: true
ipv4:
enabled: true
hostServices:
enabled: true
hostPort:
enabled: true
nodePort:
enabled: true
externalIPs:
enabled: true
hostFirewall:
enabled: true
Hack to add pod CIDR to kubespan with advertiseKubernetesNetworks: true:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: cilium-host-node-cidr
namespace: kube-system
spec:
selector:
matchLabels:
app: cilium-host-node-cidr
template:
metadata:
name: cilium-host-node-cidr
labels:
app: cilium-host-node-cidr
spec:
hostNetwork: true
tolerations:
- key: "node-role.kubernetes.io/master"
operator: Exists
- key: "node-role.kubernetes.io/control-plane"
operator: Exists
containers:
- name: cilium-host-node-cidr
image: alpine
imagePullPolicy: Always
command:
- /bin/sh
- -c
- |
apk update
apk add iproute2
handle_error() {
echo "$1"
sleep "$SLEEP_TIME"
}
echo "Watching cilium_host IP addresses..."
while :; do
# Extract all IPv4 addresses from cilium_host
ip_addresses=$(ip -4 addr show dev cilium_host |grep inet | awk '{print $2}')
# Check if any of the IP addresses match the NODE_CIDR_MASK_SIZE
echo "$ip_addresses" | grep -q "/${NODE_CIDR_MASK_SIZE}" || {
# Extract the /32 IP address if NODE_CIDR_MASK_SIZE was not found
pod_ip=$(echo "$ip_addresses" | grep "/32" | cut -d/ -f1)
if [ -z "$pod_ip" ]; then
handle_error "Couldn't extract cilium pod IP address from cilium_host interface"
continue
fi
# Add secondary IP address with the proper NODE_CIDR_MASK_SIZE
echo "cilium_host IP is $pod_ip"
ip addr add "${pod_ip}/${NODE_CIDR_MASK_SIZE}" dev cilium_host
echo "Added new cilium_host IP address with mask /${NODE_CIDR_MASK_SIZE}"
ip addr show dev cilium_host
}
sleep "$SLEEP_TIME"
done
env:
# The node cidr mask size (IPv4) to allocate pod IPs
- name: NODE_CIDR_MASK_SIZE
value: "24"
- name: SLEEP_TIME
value: "30"
securityContext:
capabilities:
add: ["NET_ADMIN"]
----------------------------------------------------------------------------------------
cilium connectivity perf --tolerations "" --namespace-labels pod-security.kubernetes.io/enforce=privileged -n kube-system --helm-release-name cilium --udp --crr --samples 3 \
--node-selector-client "kubernetes.io/hostname=io-apps-bootstrap-1" --node-selector-server "kubernetes.io/hostname=io-gpu-8tv12b3n8mss73a5"
🔥 Network Performance Test Summary - NON COLOCATED NODES (AWS->ONPREM):
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 Scenario | Node | Test | Duration | Min | Mean | Max | P50 | P90 | P99 | Transaction rate OP/s
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 pod-to-pod | same-node | TCP_CRR | 10s | 58µs | 67.04µs | 306µs | 65µs | 73µs | 98µs | 14866.02
📋 pod-to-pod | same-node | TCP_RR | 10s | 17µs | 33.9µs | 544µs | 33µs | 37µs | 50µs | 29380.88
📋 pod-to-pod | same-node | UDP_RR | 10s | 18µs | 32.77µs | 181µs | 32µs | 35µs | 48µs | 30398.39
📋 host-to-host | same-node | TCP_CRR | 10s | 67µs | 85.55µs | 381µs | 85µs | 93µs | 121µs | 11661.15
📋 host-to-host | same-node | TCP_RR | 10s | 23µs | 36.35µs | 618µs | 36µs | 40µs | 53µs | 27401.58
📋 host-to-host | same-node | UDP_RR | 10s | 21µs | 30.9µs | 210µs | 30µs | 32µs | 46µs | 32233.90
📋 pod-to-pod | other-node | TCP_CRR | 10s | 264.6ms | 265.32654ms | 266.027ms | 264.864ms | 268.918ms | 269.729ms | 3.67
📋 pod-to-pod | other-node | TCP_RR | 10s | 132.279ms | 134.5613ms | 264.645ms | 135.068ms | 139.041ms | 140ms | 7.33
📋 pod-to-pod | other-node | UDP_RR | 10s | 132.226ms | 132.69796ms | 133.179ms | 134.933ms | 138.933ms | 139.866ms | 7.50
📋 host-to-host | other-node | TCP_CRR | 10s | 264.521ms | 265.27635ms | 269.059ms | 264.864ms | 268.918ms | 269.729ms | 3.67
📋 host-to-host | other-node | TCP_RR | 10s | 132.466ms | 134.62891ms | 264.991ms | 135.068ms | 139.041ms | 140ms | 7.33
📋 host-to-host | other-node | UDP_RR | 10s | 132.281ms | 132.93875ms | 141.145ms | 135ms | 139.054ms | 140ms | 7.50
📋 pod-to-pod | same-node | TCP_CRR | 10s | 58µs | 67.35µs | 3.463ms | 65µs | 73µs | 101µs | 14799.03
📋 pod-to-pod | same-node | TCP_RR | 10s | 19µs | 33.57µs | 257µs | 33µs | 36µs | 49µs | 29675.29
📋 pod-to-pod | same-node | UDP_RR | 10s | 18µs | 32.75µs | 39.239ms | 32µs | 35µs | 48µs | 30418.80
📋 host-to-host | same-node | TCP_CRR | 10s | 67µs | 87.48µs | 29.116ms | 86µs | 97µs | 128µs | 11402.00
📋 host-to-host | same-node | TCP_RR | 10s | 22µs | 36.17µs | 215µs | 36µs | 39µs | 52µs | 27552.41
📋 host-to-host | same-node | UDP_RR | 10s | 22µs | 31.26µs | 192µs | 30µs | 35µs | 47µs | 31862.74
📋 pod-to-pod | other-node | TCP_CRR | 10s | 264.337ms | 265.24054ms | 265.849ms | 264.864ms | 268.918ms | 269.729ms | 3.67
📋 pod-to-pod | other-node | TCP_RR | 10s | 132.212ms | 134.49454ms | 265.236ms | 135.068ms | 139.041ms | 140ms | 7.34
📋 pod-to-pod | other-node | UDP_RR | 10s | 132.263ms | 132.72049ms | 134.5ms | 134.933ms | 138.933ms | 139.866ms | 7.50
📋 host-to-host | other-node | TCP_CRR | 10s | 264.701ms | 265.3847ms | 266.411ms | 264.864ms | 268.918ms | 269.729ms | 3.67
📋 host-to-host | other-node | TCP_RR | 10s | 132.197ms | 134.60853ms | 265.78ms | 135.068ms | 139.041ms | 140ms | 7.33
📋 host-to-host | other-node | UDP_RR | 10s | 132.154ms | 132.91133ms | 148.356ms | 135ms | 139.054ms | 140ms | 7.50
📋 pod-to-pod | same-node | TCP_CRR | 10s | 59µs | 69.47µs | 311µs | 66µs | 80µs | 118µs | 14344.16
📋 pod-to-pod | same-node | TCP_RR | 10s | 19µs | 33.46µs | 229µs | 33µs | 36µs | 48µs | 29773.55
📋 pod-to-pod | same-node | UDP_RR | 10s | 20µs | 32.65µs | 214µs | 32µs | 35µs | 48µs | 30515.23
📋 host-to-host | same-node | TCP_CRR | 10s | 68µs | 85.89µs | 537µs | 85µs | 93µs | 123µs | 11611.06
📋 host-to-host | same-node | TCP_RR | 10s | 25µs | 36.42µs | 192µs | 36µs | 39µs | 52µs | 27371.92
📋 host-to-host | same-node | UDP_RR | 10s | 22µs | 30.92µs | 297µs | 30µs | 33µs | 46µs | 32213.09
📋 pod-to-pod | other-node | TCP_CRR | 10s | 265.235ms | 266.09854ms | 269.077ms | 264.864ms | 268.918ms | 269.729ms | 3.66
📋 pod-to-pod | other-node | TCP_RR | 10s | 132.446ms | 134.7497ms | 265.69ms | 135.068ms | 139.041ms | 140ms | 7.32
📋 pod-to-pod | other-node | UDP_RR | 10s | 132.496ms | 133.04328ms | 134.042ms | 134.933ms | 138.933ms | 139.866ms | 7.50
📋 host-to-host | other-node | TCP_CRR | 10s | 265.256ms | 265.82441ms | 268.172ms | 264.864ms | 268.918ms | 269.729ms | 3.66
📋 host-to-host | other-node | TCP_RR | 10s | 132.648ms | 134.87827ms | 265.483ms | 135.068ms | 139.041ms | 140ms | 7.32
📋 host-to-host | other-node | UDP_RR | 10s | 132.486ms | 133.12277ms | 139.521ms | 134.933ms | 138.933ms | 139.866ms | 7.50
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
📋 Scenario | Node | Test | Duration | Throughput Mb/s
----------------------------------------------------------------------------------------
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 12374.78
📋 pod-to-pod | same-node | UDP_STREAM | 10s | 1922.25
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 46342.63
📋 pod-to-pod | same-node | UDP_STREAM_MULTI | 10s | 7505.86
📋 host-to-host | same-node | TCP_STREAM | 10s | 20529.12
📋 host-to-host | same-node | UDP_STREAM | 10s | 1393.09
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 79555.81
📋 host-to-host | same-node | UDP_STREAM_MULTI | 10s | 5320.73
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 130.48
📋 pod-to-pod | other-node | UDP_STREAM | 10s | 403.64
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 462.46
📋 pod-to-pod | other-node | UDP_STREAM_MULTI | 10s | 522.04
📋 host-to-host | other-node | TCP_STREAM | 10s | 117.54
📋 host-to-host | other-node | UDP_STREAM | 10s | 423.49
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 491.17
📋 host-to-host | other-node | UDP_STREAM_MULTI | 10s | 548.12
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 12309.88
📋 pod-to-pod | same-node | UDP_STREAM | 10s | 1926.65
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 46042.69
📋 pod-to-pod | same-node | UDP_STREAM_MULTI | 10s | 7527.96
📋 host-to-host | same-node | TCP_STREAM | 10s | 20760.76
📋 host-to-host | same-node | UDP_STREAM | 10s | 1373.03
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 79836.82
📋 host-to-host | same-node | UDP_STREAM_MULTI | 10s | 5281.56
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 127.08
📋 pod-to-pod | other-node | UDP_STREAM | 10s | 376.28
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 297.87
📋 pod-to-pod | other-node | UDP_STREAM_MULTI | 10s | 525.94
📋 host-to-host | other-node | TCP_STREAM | 10s | 119.80
📋 host-to-host | other-node | UDP_STREAM | 10s | 431.03
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 458.48
📋 host-to-host | other-node | UDP_STREAM_MULTI | 10s | 560.72
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 12186.17
📋 pod-to-pod | same-node | UDP_STREAM | 10s | 1931.67
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 45567.46
📋 pod-to-pod | same-node | UDP_STREAM_MULTI | 10s | 7430.05
📋 host-to-host | same-node | TCP_STREAM | 10s | 20255.60
📋 host-to-host | same-node | UDP_STREAM | 10s | 1385.20
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 79924.62
📋 host-to-host | same-node | UDP_STREAM_MULTI | 10s | 5241.39
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 117.64
📋 pod-to-pod | other-node | UDP_STREAM | 10s | 376.95
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 488.13
📋 pod-to-pod | other-node | UDP_STREAM_MULTI | 10s | 542.54
📋 host-to-host | other-node | TCP_STREAM | 10s | 117.60
📋 host-to-host | other-node | UDP_STREAM | 10s | 408.21
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 471.71
📋 host-to-host | other-node | UDP_STREAM_MULTI | 10s | 583.24
----------------------------------------------------------------------------------------
🔥 Network Performance Test Summary - COLOCATED NODES (AWS->AWS):
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 Scenario | Node | Test | Duration | Min | Mean | Max | P50 | P90 | P99 | Transaction rate OP/s
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 pod-to-pod | same-node | TCP_CRR | 10s | 94µs | 151.37µs | 16.201ms | 134µs | 195µs | 406µs | 6581.04
📋 pod-to-pod | same-node | TCP_RR | 10s | 31µs | 50.48µs | 6.289ms | 48µs | 63µs | 130µs | 19649.36
📋 pod-to-pod | same-node | UDP_RR | 10s | 30µs | 49.74µs | 5.088ms | 47µs | 63µs | 127µs | 19931.64
📋 host-to-host | same-node | TCP_CRR | 10s | 101µs | 160.46µs | 5.724ms | 145µs | 206µs | 455µs | 6210.43
📋 host-to-host | same-node | TCP_RR | 10s | 32µs | 49.27µs | 21.005ms | 44µs | 62µs | 121µs | 20122.15
📋 host-to-host | same-node | UDP_RR | 10s | 31µs | 49.06µs | 4.848ms | 45µs | 62µs | 122µs | 20200.67
📋 pod-to-pod | other-node | TCP_CRR | 10s | 762µs | 1.15455ms | 18.837ms | 1.057ms | 1.413ms | 2.842ms | 865.36
📋 pod-to-pod | other-node | TCP_RR | 10s | 305µs | 466.07µs | 7.793ms | 435µs | 536µs | 1.211ms | 2142.90
📋 pod-to-pod | other-node | UDP_RR | 10s | 299µs | 451.41µs | 16.472ms | 418µs | 526µs | 1.162ms | 2212.38
📋 host-to-host | other-node | TCP_CRR | 10s | 714µs | 1.11194ms | 8.506ms | 996µs | 1.361ms | 3.5ms | 898.60
📋 host-to-host | other-node | TCP_RR | 10s | 295µs | 461.86µs | 7.879ms | 416µs | 538µs | 1.471ms | 2162.46
📋 host-to-host | other-node | UDP_RR | 10s | 292µs | 430.47µs | 12.183ms | 400µs | 501µs | 1.091ms | 2320.17
📋 pod-to-pod | same-node | TCP_CRR | 10s | 95µs | 158.23µs | 12.025ms | 135µs | 202µs | 470µs | 6298.03
📋 pod-to-pod | same-node | TCP_RR | 10s | 31µs | 50.67µs | 4.959ms | 49µs | 64µs | 118µs | 19571.78
📋 pod-to-pod | same-node | UDP_RR | 10s | 30µs | 48.64µs | 5.806ms | 47µs | 62µs | 116µs | 20379.56
📋 host-to-host | same-node | TCP_CRR | 10s | 107µs | 162.46µs | 13.512ms | 143µs | 203µs | 543µs | 6134.84
📋 host-to-host | same-node | TCP_RR | 10s | 31µs | 49.95µs | 9.565ms | 46µs | 64µs | 116µs | 19844.92
📋 host-to-host | same-node | UDP_RR | 10s | 32µs | 53.8µs | 9.315ms | 50µs | 68µs | 140µs | 18435.33
📋 pod-to-pod | other-node | TCP_CRR | 10s | 779µs | 1.10845ms | 7.736ms | 1.032ms | 1.328ms | 2.68ms | 901.38
📋 pod-to-pod | other-node | TCP_RR | 10s | 308µs | 453.37µs | 6.071ms | 419µs | 526µs | 1.196ms | 2203.14
📋 pod-to-pod | other-node | UDP_RR | 10s | 303µs | 473.46µs | 19.026ms | 419µs | 543µs | 1.704ms | 2109.48
📋 host-to-host | other-node | TCP_CRR | 10s | 739µs | 1.08138ms | 14.969ms | 993µs | 1.326ms | 2.566ms | 923.96
📋 host-to-host | other-node | TCP_RR | 10s | 293µs | 429.17µs | 11.309ms | 396µs | 491µs | 1.155ms | 2327.37
📋 host-to-host | other-node | UDP_RR | 10s | 288µs | 429.47µs | 8.627ms | 398µs | 493µs | 1.088ms | 2325.36
📋 pod-to-pod | same-node | TCP_CRR | 10s | 95µs | 159.08µs | 19.388ms | 135µs | 198µs | 479µs | 6263.89
📋 pod-to-pod | same-node | TCP_RR | 10s | 31µs | 50.82µs | 14.357ms | 48µs | 63µs | 126µs | 19507.35
📋 pod-to-pod | same-node | UDP_RR | 10s | 30µs | 48.42µs | 6.859ms | 46µs | 61µs | 115µs | 20470.00
📋 host-to-host | same-node | TCP_CRR | 10s | 98µs | 166.47µs | 16.335ms | 144µs | 204µs | 506µs | 5986.98
📋 host-to-host | same-node | TCP_RR | 10s | 32µs | 48.7µs | 5.069ms | 45µs | 62µs | 110µs | 20346.95
📋 host-to-host | same-node | UDP_RR | 10s | 32µs | 49.25µs | 4.462ms | 45µs | 62µs | 123µs | 20121.02
📋 pod-to-pod | other-node | TCP_CRR | 10s | 756µs | 1.16556ms | 12.169ms | 1.052ms | 1.434ms | 3.622ms | 857.29
📋 pod-to-pod | other-node | TCP_RR | 10s | 308µs | 474.68µs | 13.619ms | 421µs | 535µs | 1.804ms | 2104.15
📋 pod-to-pod | other-node | UDP_RR | 10s | 305µs | 452.77µs | 12.858ms | 420µs | 526µs | 1.15ms | 2205.79
📋 host-to-host | other-node | TCP_CRR | 10s | 731µs | 1.06169ms | 9.363ms | 980µs | 1.257ms | 2.792ms | 940.73
📋 host-to-host | other-node | TCP_RR | 10s | 289µs | 440.75µs | 10.215ms | 403µs | 506µs | 1.168ms | 2265.83
📋 host-to-host | other-node | UDP_RR | 10s | 298µs | 446.79µs | 26.481ms | 412µs | 515µs | 1.081ms | 2235.16
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
📋 Scenario | Node | Test | Duration | Throughput Mb/s
----------------------------------------------------------------------------------------
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 8197.07
📋 pod-to-pod | same-node | UDP_STREAM | 10s | 739.78
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 11374.82
📋 pod-to-pod | same-node | UDP_STREAM_MULTI | 10s | 1033.50
📋 host-to-host | same-node | TCP_STREAM | 10s | 16969.26
📋 host-to-host | same-node | UDP_STREAM | 10s | 732.87
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 23484.88
📋 host-to-host | same-node | UDP_STREAM_MULTI | 10s | 810.38
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 1526.48
📋 pod-to-pod | other-node | UDP_STREAM | 10s | 364.91
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 1680.45
📋 pod-to-pod | other-node | UDP_STREAM_MULTI | 10s | 518.74
📋 host-to-host | other-node | TCP_STREAM | 10s | 1801.35
📋 host-to-host | other-node | UDP_STREAM | 10s | 420.75
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 1809.80
📋 host-to-host | other-node | UDP_STREAM_MULTI | 10s | 428.54
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 8220.26
📋 pod-to-pod | same-node | UDP_STREAM | 10s | 888.74
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 11466.74
📋 pod-to-pod | same-node | UDP_STREAM_MULTI | 10s | 845.84
📋 host-to-host | same-node | TCP_STREAM | 10s | 17271.49
📋 host-to-host | same-node | UDP_STREAM | 10s | 646.52
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 23601.75
📋 host-to-host | same-node | UDP_STREAM_MULTI | 10s | 881.83
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 1563.01
📋 pod-to-pod | other-node | UDP_STREAM | 10s | 346.93
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 1693.54
📋 pod-to-pod | other-node | UDP_STREAM_MULTI | 10s | 512.22
📋 host-to-host | other-node | TCP_STREAM | 10s | 1877.99
📋 host-to-host | other-node | UDP_STREAM | 10s | 393.55
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 1851.96
📋 host-to-host | other-node | UDP_STREAM_MULTI | 10s | 545.90
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 8175.70
📋 pod-to-pod | same-node | UDP_STREAM | 10s | 874.48
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 11698.93
📋 pod-to-pod | same-node | UDP_STREAM_MULTI | 10s | 855.51
📋 host-to-host | same-node | TCP_STREAM | 10s | 17208.02
📋 host-to-host | same-node | UDP_STREAM | 10s | 709.62
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 23487.27
📋 host-to-host | same-node | UDP_STREAM_MULTI | 10s | 679.89
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 1553.33
📋 pod-to-pod | other-node | UDP_STREAM | 10s | 360.14
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 1712.79
📋 pod-to-pod | other-node | UDP_STREAM_MULTI | 10s | 524.76
📋 host-to-host | other-node | TCP_STREAM | 10s | 1813.94
📋 host-to-host | other-node | UDP_STREAM | 10s | 436.07
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 1803.77
📋 host-to-host | other-node | UDP_STREAM_MULTI | 10s | 539.33
----------------------------------------------------------------------------------------
Configuration 1 – Standard Cilium (VXLAN)
- Cross-node latency: ~287ms (TCP_CRR), ~144ms (TCP_RR)
- Cross-node throughput: 105–430 Mb/s
- Same-node performance: Excellent (14–29k ops/s, 12–79 Gb/s throughput)
Configuration 2 – Native Routing
- Cross-node latency: ~265ms (TCP_CRR), ~134ms (TCP_RR)
- Cross-node throughput: 117–583 Mb/s (modest but noticeable improvements)
- Same-node performance: Comparable to VXLAN setup
4. Key Observations
Latency: Native routing consistently shaved off 7–20ms across nodes.
Throughput: Gains were modest, but improvements were more visible in UDP scenarios.
Simplicity: Removing VXLAN reduces encapsulation overhead and makes the datapath more transparent.
5. Conclusion
Native routing in Cilium does provide measurable improvements in hybrid setups, lower latency, slightly better throughput, and a cleaner datapath.
That said, the improvements are incremental rather than game-changing. Given the complexity of the workaround required, I don’t consider it production ready for now.
The good news is that the Sidero community is actively enhancing KubeSpan, and future releases may support native routing out of the box. If that happens, we’ll be able to combine the security and NAT traversal of KubeSpan with the performance benefits of native routing, without custom hacks.
Top comments (0)