Cross-posted from otherthings.cloud, where the rest of my homelab notes live.
The last post ended by saying NoOverlay mode was next. This is that post.
In OpenShift 4.22, a user-defined network can drop the overlay entirely. Pod traffic between nodes goes out as plain IP. Each node's pod subnet is advertised over BGP, and the network routes it like any other prefix.
The documentation gives one reason for it:
You can use no-overlay mode to route layer 3 pod traffic directly over the underlay network with BGP, which reduces encapsulation overhead and improves east-west performance.
Performance is the one thing I can't measure honestly. My cluster is three nested VMs on one Proxmox host. So this post leaves speed for later and asks the other questions: what goes on the wire, what the network sees, and what it took to get there.
TL;DR
With BGP on the node's primary NIC, a pod reaches the outside world with its own IP, and the outside world reaches it back, without any change to host settings. My first build used a second NIC for BGP, and that version didn't round-trip. And two NoOverlay networks stayed isolated from each other, even though the fabric would happily have routed between them.
I hope the diagram is readable at this size. If it isn't, click it to open it full size.
This is the final layout, with BGP on the primary NIC. VLAN 30 is idle here, but my first build peered over it, as the next sections show. The packets in "On the wire" were captured during that first build.
The setup: one cluster, both paths
No-overlay mode is Technology Preview in 4.22. Using it requires the TechPreviewNoUpgrade feature set, which can't be undone and blocks minor-version updates. I didn't want that on the cluster from the earlier posts, so I built a separate three-node cluster, ocp-nooverlay, for this one.
The default pod network stayed on Geneve. Only a new CUDN uses NoOverlay. That gives one cluster with both paths side by side, and every other variable is the same.
The docs show a noOverlayOptions block on the CUDN. On my build, with the feature gate on, that field doesn't exist:
error: field "noOverlayOptions" does not exist
So the CUDN carries transport: NoOverlay and nothing else. Its first status was not what I expected:
TransportAccepted False No RouteAdvertisements CR is advertising the pod networks.
The transport isn't accepted until something advertises the network. Once the RouteAdvertisements existed, it changed:
TransportAccepted True Transport has been configured as 'no-overlay'.
First build: BGP on a second NIC
Each node has two NICs. The first is the machine network (VLAN 20, behind br-ex). The second sits on VLAN 30, where spine1, the route reflector from the earlier posts, lives. The EVPN cluster peered with spine1 over that second NIC, so I copied the layout.
The routes arrived as planned. Each node learned the other two nodes' pod /24s from spine1, with next hops on VLAN 30:
== ocp-nooverlay-cp1
192.168.52.0/24 nhid 378 via 192.168.30.50 dev enp6s19 proto bgp metric 20
192.168.53.0/24 nhid 376 via 192.168.30.51 dev enp6s19 proto bgp metric 20
Pod to pod across nodes worked:
64 bytes from 192.168.52.5: icmp_seq=1 ttl=60 time=20.6 ms
On the wire
I expected that ping on the VLAN 30 NIC, enp6s19, since that's where the routes point. The capture there caught nothing: a 24-byte file, just the pcap header.
The packets were on the other NIC. A capture on all interfaces found them leaving through enp6s18, the machine network:
22:35:49.274114 enp6s18 Out ifindex 2 52:54:00:00:20:51 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 62, id 4202, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.54.5 > 192.168.52.5: ICMP echo request, id 9, seq 1, length 64
It's plain ICMP, pod IP to pod IP, with no outer header. The IP packet is 84 bytes.
For comparison, here is a ping between two pods on the default Geneve network, on the same NIC:
22:33:10.419329 enp6s18 Out ifindex 2 52:54:00:00:20:51 ethertype IPv4 (0x0800), length 162: (tos 0x0, ttl 64, id 13416, offset 0, flags [DF], proto UDP (17), length 142)
192.168.20.51.63217 > 192.168.20.52.6081: Geneve, Flags [C], vni 0xff0003, proto TEB (0x6558), options [class Open Virtual Networking (OVN) (0x102) type 0x80(C) len 8]
0a:58:64:58:00:04 > 0a:58:64:58:00:03, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 26285, offset 0, flags [DF], proto ICMP (1), length 84)
10.130.0.146 > 10.128.0.173: ICMP echo request, id 8, seq 1, length 64
The same 84-byte ICMP packet travels inside a 142-byte UDP packet to port 6081. That's 58 bytes of encapsulation: the 50 bytes VXLAN costs, plus an 8-byte OVN option. NoOverlay adds nothing.
The MTU didn't change
With no encapsulation, I expected the pod MTU to grow to the underlay's 1500. It didn't:
3: ovn-udn1@if157: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP mode DEFAULT group default qlen 1000
A 1500-byte ping with don't-fragment set never left the pod:
ping: local error: message too long, mtu=1400
The NoOverlay CUDN took the cluster's 1400. The CUDN spec has an mtu field, but I haven't tested whether a NoOverlay network accepts 1500 there.
Off the cluster: out un-NATed, nothing back
The next question was what the network sees when a pod talks to something outside the cluster. I pinged spine1 from a NoOverlay pod and captured on spine1:
18:12:24.444219 48:a9:8a:38:16:62 > bc:24:11:3a:b2:ba, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 61, id 32720, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.54.5 > 192.168.30.2: ICMP echo request, id 12, seq 1, length 64
18:12:24.444624 bc:24:11:3a:b2:ba > 52:54:00:00:30:49, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 65249, offset 0, flags [none], proto ICMP (1), length 84)
192.168.30.2 > 192.168.54.5: ICMP echo reply, id 12, seq 1, length 64
The source is 192.168.54.5, the pod's own address, not the node's. That's what I wanted to see.
spine1 replied, too. But the pod never got the reply:
5 packets transmitted, 0 received, 100% packet loss, time 4072ms
Pinging the pod from spine1 failed the same way.
The two directions took different paths. The request left the node by its default route, via 192.168.20.1 dev br-ex, on the machine network, and was routed to spine1 from there. The reply followed spine1's BGP route, which points at the node's VLAN 30 address. So the reply arrived on enp6s19.
OVN-Kubernetes doesn't manage enp6s19, and forwarding is off on it:
net.ipv4.conf.enp6s19.forwarding = 0
The node knows where the pod is. It just won't forward a packet arriving on that NIC:
[root@ocp-nooverlay-cp1 /]# ip route get 192.168.54.5 from 192.168.30.2 iif enp6s19
RTNETLINK answers: No route to host
Host settings on that NIC can change this. I didn't want a design that depends on tuning the host. So I moved the BGP session instead.
Peering on the primary NIC
I gave spine1 a leg on VLAN 20 and pointed the cluster's FRRConfiguration at that address instead. Nothing else in the cluster changed.
FRR-K8s used each node's VLAN 20 address as the next hop with no extra configuration:
*>i192.168.52.0/24 192.168.20.52 0 100 0 i
*>i192.168.53.0/24 192.168.20.53 0 100 0 i
*>i192.168.54.0/24 192.168.20.51 0 100 0 i
Then the same test again, from both ends, captured on spine1's VLAN 20 NIC, ens19.
The pod pings spine1:
13:32:11.370234 ens19 In ifindex 3 52:54:00:00:20:51 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 62, id 53389, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.54.5 > 192.168.20.25: ICMP echo request, id 44, seq 1, length 64
13:32:11.370396 ens19 Out ifindex 3 52:54:00:00:20:25 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 64, id 22216, offset 0, flags [none], proto ICMP (1), length 84)
192.168.20.25 > 192.168.54.5: ICMP echo reply, id 44, seq 1, length 64
spine1 pings the pod at its own address:
13:32:34.108343 ens19 Out ifindex 3 52:54:00:00:20:25 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 64, id 25468, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.20.25 > 192.168.54.5: ICMP echo request, id 4, seq 1, length 64
13:32:34.110169 ens19 In ifindex 3 52:54:00:00:20:51 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 62, id 7458, offset 0, flags [none], proto ICMP (1), length 84)
192.168.54.5 > 192.168.20.25: ICMP echo reply, id 4, seq 1, length 64
Both directions were 5/5. There's no NodePort, Service or LoadBalancer in the path. The pod is just an address on the network.
A pod on the default Geneve network, pinging the same target over the same wire, arrives as its node:
13:32:38.383135 ens19 In ifindex 3 52:54:00:00:20:51 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 62, id 32935, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.20.51 > 192.168.20.25: ICMP echo request, id 48889, seq 1, length 64
That's the whole difference between the two networks, as spine1 sees it: 192.168.54.5 against 192.168.20.51.
Afterwards I read the node's settings again. Nothing had been changed:
net.ipv4.ip_forward = 0
net.ipv4.conf.enp6s19.forwarding = 0
net.ipv4.conf.enp6s19.rp_filter = 1
net.ipv4.conf.enp6s18.forwarding = 0
net.ipv4.conf.br-ex.forwarding = 1
Forwarding is on only for br-ex, the interface OVN-Kubernetes manages. With BGP on the primary NIC, traffic arrives on br-ex, so that's all it needs.
Isolation still holds
One question was still bothering me. With Geneve, two user-defined networks are kept apart, and I'd measured that in an earlier lab: pods on different UDNs can't reach each other. But with NoOverlay there is no tunnel to keep them apart. Every network is just routes.
So I created a second NoOverlay CUDN, B, next to A. spine1 now holds both networks' subnets in one table:
*>i192.168.52.0/24 192.168.20.52 0 100 0 i
*>i192.168.53.0/24 192.168.20.53 0 100 0 i
*>i192.168.54.0/24 192.168.20.51 0 100 0 i
*>i192.168.56.0/24 192.168.20.52 0 100 0 i
*>i192.168.57.0/24 192.168.20.53 0 100 0 i
*>i192.168.58.0/24 192.168.20.51 0 100 0 i
The node had a route too. On cp1, the kernel's table for network A knows how to reach B's pod on cp2:
192.168.56.4 from 192.168.54.5 via 192.168.20.52 dev br-ex table 1067
Both controls passed first: B's pods reached each other, and spine1 reached B's pod. Then A to B:
PING 192.168.56.4 (192.168.56.4) 56(84) bytes of data.
--- 192.168.56.4 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4078ms
A to B failed on the same node and across nodes, and B to A failed too. Every case lost every packet, silently. That's the same result, with the same silence, as Geneve UDNs.
The kernel route was never used. In a capture on cp1, the packets appear only on pod A's own port, and never on enp6s18. The capture on cp2 was empty:
14:41:05.545535 ? P ifindex 157 0a:58:c0:a8:36:05 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 64, id 16698, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.54.5 > 192.168.56.4: ICMP echo request, id 52, seq 1, length 64
So the packet is dropped inside the first node, before it reaches any routing table.
The rule that drops it
NOTE: The commands for these traces and lookups, so you can find the same rules on your own cluster, are in
isolation-trace.md.
To find out where, I used ovn-trace. OVN turns its configuration (switches, routers, ACLs) into a long list of numbered rules. ovn-trace runs a made-up packet through that list without sending anything, and prints each rule the packet matches, in order. I gave it a ping from A's pod on cp1 to B's pod.
Most of the rules it passed are routine. This is the one that stopped it:
20. ls_in_acl_after_lb_eval (northd.c:7608): reg8[30..31] == 0 && reg0[9] == 1 && ((ip4.src == $a10109792604843350142 && ip4.dst == $a10109792604843350142)), priority 2050, uuid a5d41540
reg8[17] = 1;
The part that matters is in the double brackets. If the source and the destination are both in $a10109792604843350142, the rule sets reg8[17], which marks the packet to be dropped. The trace ends at the next rule. Nothing after it forwards the packet.
$a10109792604843350142 is an address set: a named list of subnets that rules can refer to. OVN's database shows what's in it:
addresses : ["192.168.52.0/22", "192.168.56.0/22"]
Those are the whole subnets of A and B. Each network is a /22, and spine1's table above shows the /24 slice each node takes from it. So the rule drops any packet whose source and destination are both on the list of advertised networks.
That would drop A to A too. It doesn't, because a rule with higher priority wins. When I traced A pod to A pod, this rule matched first and let the packet through:
20. ls_in_acl_after_lb_eval (northd.c:7480): reg8[30..31] == 0 && (((ip4.src == 192.168.52.0/22) && (ip4.dst == 192.168.52.0/22))), priority 2100, uuid 72ab9e68
It matches only traffic that stays inside A's own subnet, and at 2100 it beats the drop rule at 2050. Between them, the two rules allow traffic within a network and drop traffic between networks.
In OVN's database, both the address set and the rule are owned by AdvertisedNetwork, not by anything named for NoOverlay. From that I'd guess the same rule separates any advertised UDNs, Geneve or EVPN included. I haven't tested that.
Conclusion
Peer on the primary NIC, and NoOverlay does what it says: plain packets, the pod's own address on the network, and nothing to tune on the host. Isolation between networks survives the loss of the tunnel, because it never depended on the tunnel.
The part I couldn't measure is the reason the feature exists: performance. That needs real hardware, and it's the next part of this series.
Everything needed to build it is in 2026-10-openshift-nooverlay/.
Top comments (0)