DEV Community: Marvin

Live-Migrating an OpenShift Virtualization VM Across an EVPN MAC-VRF

Marvin — Fri, 30 Oct 2026 21:03:01 +0000

Cross-posted from otherthings.cloud, where the rest of my homelab notes live.

The last post joined the OpenShift 4.22 cluster to the fabric through the IP-VRF: pods on their own subnet, routed to host-a. It ended with the MAC-VRF side untested. This post tests that side.

A MAC-VRF stretches a Layer 2 segment across the EVPN fabric for east-west bridged traffic and workloads such as VM live migration.

NOTE: See the OVN-Kubernetes EVPN documentation for the MAC-VRF definition.

I wanted a VM on OpenShift to sit on the same subnet as host-a and host-b: 192.168.40.0/24. The two leaves already stretch this subnet between them as VNI 100 in my lab.

Then I wanted to live-migrate the VM to another node (cp1 -> cp2) while host-b continuously pinged it, and see what the fabric made of the move.

The VM kept its IP and its MAC. The fabric treated the migration as an EVPN MAC move, with a sequence number increasing each time. host-b lost two pings during the migration.

There was also a subnet-planning consequence that I found later in the test.


                    spine1  192.168.30.2
                    (BGP route reflector)
         ┌───────────────┬────────┬────────┬───────────┐
         │               │        │        │           │
      leaf1           leaf2     cp1      cp2          cp3
   192.168.30.11   .30.12    .31.21   .31.22       .31.23
      VTEP           VTEP     VTEP     VTEP         VTEP
         │              │        └────────┬──────-─────┘
     host-a         host-b                │
    .40.10          .40.11              vm-a1
         │              │              .40.32
         └──────────────┴────────────────┘
                192.168.40.0/24
                   VNI 100

                    cp1  ─────→  cp2
                         VM migration

   cluster nodes as VTEPs: cp1 192.168.31.21 · cp2 .22 · cp3 .23
      vm-a1 192.168.40.32 — moves between cp1 and cp2

vm-a1 starts on worker cp1 and is live-migrated to worker cp2, while host-b continuously pings vm-a1:192.168.40.32.

The network

The VM's network is a ClusterUserDefinedNetwork with topology: Layer2 and transport: EVPN. In other words, this creates a Layer 2 primary CUDN across the EVPN network as MAC-VRF:

    evpn:
      vtep: evpn-vtep
      macVRF:
        vni: 100
        routeTarget: "65000:100"
    layer2:
      role: Primary
      subnets:               ["192.168.40.0/24"]
      infrastructureSubnets: ["192.168.40.0/29"]
      defaultGatewayIPs:     ["192.168.40.1"]
      reservedSubnets:
        - "192.168.40.8/29"
        - "192.168.40.16/28"
      ipam:
        mode: Enabled
        lifecycle: Persistent
      mtu: 1400

Three parts of it matter here.

macVRF.vni: 100 is the number the leaves already use for this segment. The VNI is what identifies an L2 segment across the fabric. Using the same number puts the VM in the same broadcast domain as host-a and host-b.

The subnet lists are a fence. By default OVN-Kubernetes allocates from the whole /24, but .1, .10 and .11 already belong to the fabric. infrastructureSubnets holds the gateway, and reservedSubnets keeps .8 to .31 away from the allocator. VMs get .32 and up. The spec is immutable, so the fence cannot be widened later.

ipam.lifecycle: Persistent is what should keep the VM's address across a migration. Whether it does is one of the things I checked.

There is no ipVRF. My first design had one, using VNI 101, the fabric's tenant VRF. The API accepted the object, and then this appeared in its NetworkCreated condition:

        "message": "failed to allocate EVPN IDs: failed to reserve VNIs: VNI 101 on VTEP \"evpn-vtep\" is already reserved by network \"cudn-tenant-a\"",

cudn-tenant-a is the network from the last post, and it already holds 101. A VNI can be reserved by only one network on a VTEP. So this network is L2 only.

For this test, that costs nothing. The VM reaches host-a and host-b by bridging, and bridging needs no VRF. It does mean the VM's segment has no routed path to anything off the subnet.

The VM runs CentOS Stream 9 with the l2bridge interface binding. Its disk is an RWX PVC, because live migration needs shared storage. Before anything moved, the VMI already reported LIVE-MIGRATABLE True.

Everything needed to build it — the four manifests in apply order, and the commands for watching a migration — is in other-things-lab/2026-09-openshift-virt-macvrf.

Bridged, not routed

From the VM's console, to host-b:

[cloud-user@vm-a1 ~]$ ping -c 5 192.168.40.11
PING 192.168.40.11 (192.168.40.11) 56(84) bytes of data.
64 bytes from 192.168.40.11: icmp_seq=1 ttl=64 time=4.39 ms
64 bytes from 192.168.40.11: icmp_seq=2 ttl=64 time=1.58 ms
64 bytes from 192.168.40.11: icmp_seq=3 ttl=64 time=1.19 ms
64 bytes from 192.168.40.11: icmp_seq=4 ttl=64 time=1.25 ms
64 bytes from 192.168.40.11: icmp_seq=5 ttl=64 time=1.29 ms

--- 192.168.40.11 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 1.194/1.939/4.387/1.230 ms

ttl=64 means no router is in the path. Linux sends with 64, and every router hop takes one off.

On leaf1, during the same test to host-a:

[root@leaf1 ~]# tcpdump -ni ens18 'udp port 4789' -c 10 -v
dropped privs to tcpdump
tcpdump: listening on ens18, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:31:59.271470 IP (tos 0x0, ttl 64, id 61870, offset 0, flags [none], proto UDP (17), length 134)
    192.168.31.21.57978 > 192.168.30.11.vxlan: VXLAN, flags [I] (0x08), vni 100
IP (tos 0x0, ttl 64, id 50680, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.40.32 > 192.168.40.10: ICMP echo request, id 2, seq 1, length 64
13:31:59.272963 IP (tos 0x0, ttl 64, id 17652, offset 0, flags [none], proto UDP (17), length 134)
    192.168.30.11.44658 > 192.168.31.21.vxlan: VXLAN, flags [I] (0x08), vni 100
IP (tos 0x0, ttl 64, id 54707, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.40.10 > 192.168.40.32: ICMP echo reply, id 2, seq 1, length 64

The request arrives in VNI 100, from 192.168.31.21, which is cp1's VTEP. The reply goes back the same way. The VM is on the fabric's segment itself, not on a network routed to it.

Before the VM migration

The VM's MAC is 0a:58:c0:a8:28:20. The last four bytes are 192.168.40.32 in hex, because OVN-Kubernetes derives the MAC from the IP. So if the IP survives the migration, the MAC should survive too.

On spine1, the VM has two Type-2 routes, one for the MAC alone and one for MAC plus IP:

[root@spine1 ~]# vtysh -c 'show bgp l2vpn evpn route type macip' | grep -i -A3 '0a:58:c0:a8:28:20'
 *>i[2]:[0]:[48]:[0a:58:c0:a8:28:20]
                    192.168.31.21                 100      0 i
                    RT:65000:100 ET:8
 *>i[2]:[0]:[48]:[0a:58:c0:a8:28:20]:[32]:[192.168.40.32]
                    192.168.31.21                 100      0 i
                    RT:65000:100 ET:8

Both have next hop 192.168.31.21, which is cp1.

On leaf2, which is where host-b's traffic enters the fabric:

[root@leaf2 ~]# vtysh -c 'show evpn mac vni 100 mac 0a:58:c0:a8:28:20'
MAC: 0a:58:c0:a8:28:20
 Remote VTEP: 192.168.31.21
 Sync-info: neigh#: 0
 Local Seq: 0 Remote Seq: 0
 Uptime: 00:14:20
 Neighbors:
    192.168.40.32 Active

The Remote Seq: 0 is the MAC Mobility sequence number. RFC 7432 describes this sequence number as part of the mechanism used to detect MAC moves. When a MAC moves, the new advertisement can carry a higher sequence number, allowing other VTEPs to prefer the newer location.

The alternative would be for the cluster to withdraw the old route and advertise a fresh one. In that case the sequence number would stay at 0.

The kernel forwarding entry on leaf2 is also worth a look:

[root@leaf2 ~]# bridge fdb show | grep -i '0a:58:c0:a8:28:20'
0a:58:c0:a8:28:20 dev vxlan100 vlan 1 extern_learn master br100
0a:58:c0:a8:28:20 dev vxlan100 extern_learn master br100
0a:58:c0:a8:28:20 dev vxlan100 dst 192.168.31.21 self extern_learn

extern_learn means FRR installed the entry from BGP. The leaf did not learn the VM's MAC from traffic.

The important point before migration is therefore:
leaf2 currently knows that MAC 0a:58:c0:a8:28:20 is reachable through VTEP 192.168.31.21 (cp1).

The VM live migration

I migrated the VM in both directions: first from cp1 to cp2, and then back from cp2 to cp1. I'm quoting the second run, because its ping output is complete text and leaf2 wrote its capture to a file. The first run behaved the same way, but its ping summary survived only as a screenshot.

host-b ran ping -D -O -i 0.2 192.168.40.32. -D adds a timestamp to each line, and -O prints a line for every probe that gets no reply:

[1790191720.254543] 64 bytes from 192.168.40.32: icmp_seq=296 ttl=64 time=2.39 ms
[1790191720.454958] 64 bytes from 192.168.40.32: icmp_seq=297 ttl=64 time=1.99 ms
[1790191720.654829] 64 bytes from 192.168.40.32: icmp_seq=298 ttl=64 time=1.41 ms
[1790191721.060312] no answer yet for icmp_seq=299   ← migration
[1790191721.268299] no answer yet for icmp_seq=300
[1790191721.273476] 64 bytes from 192.168.40.32: icmp_seq=301 ttl=64 time=2.24 ms
[1790191721.470693] 64 bytes from 192.168.40.32: icmp_seq=302 ttl=64 time=1.77 ms

The two missing replies (icmp_seq=299 and 300) are the only packet-loss window during the migration.
From host-b's side, 0.619 s passed between the reply to 298 and the reply to 301.

leaf2 saw the same probes. Its clock is on local time, five hours behind the UTC timestamps above:

14:28:40.600748 IP 192.168.30.12.44524 > 192.168.31.22.4789: VXLAN, flags [I] (0x08), vni 100   IP 192.168.40.11 > 192.168.40.32: ICMP echo request, id 5, seq 298, length 64
14:28:40.601534 IP 192.168.31.22.53542 > 192.168.30.12.4789: VXLAN, flags [I] (0x08), vni 100   IP 192.168.40.32 > 192.168.40.11: ICMP echo reply, id 5, seq 298, length 64
14:28:40.801638 IP 192.168.30.12.44524 > 192.168.31.22.4789: VXLAN, flags [I] (0x08), vni 100   IP 192.168.40.11 > 192.168.40.32: ICMP echo request, id 5, seq 299, length 64
14:28:41.007759 IP 192.168.30.12.44524 > 192.168.31.22.4789: VXLAN, flags [I] (0x08), vni 100   IP 192.168.40.11 > 192.168.40.32: ICMP echo request, id 5, seq 300, length 64
14:28:41.215730 IP 192.168.30.12.44524 > 192.168.31.21.4789: VXLAN, flags [I] (0x08), vni 100   IP 192.168.40.11 > 192.168.40.32: ICMP echo request, id 5, seq 301, length 64
14:28:41.217416 IP 192.168.31.21.49680 > 192.168.30.12.4789: VXLAN, flags [I] (0x08), vni 100   IP 192.168.40.32 > 192.168.40.11: ICMP echo reply, id 5, seq 301, length 64

Seq 298 went to 192.168.31.22, cp2, and was answered.

Seq 299 and 300 also went to cp2, but the VM was no longer answering there.

Seq 301 went to 192.168.31.21, cp1, and was answered 1.7 ms later.

So the fabric didn't drop the lost probes. It delivered them to where the VM had been. Between 41.007759 and 41.215730, leaf2 changed one destination. Nothing was flooded, and no packet went to both nodes.

As leaf2 saw it, 616 ms passed between the last reply from cp2 and the first reply from cp1. The VM was unreachable for somewhere between about 0.2 s and 0.6 s.

The migration's own timeline, in UTC:

❯ oc get vmim kubevirt-migrate-vm-bh22k -n vm-a -o jsonpath='{range.status.phaseTransitionTimestamps[*]}{.phase}{"\t"}{.phaseTransitionTimestamp}{"\n"}{end}'
Pending    2026-09-23T19:28:36Z
Scheduling    2026-09-23T19:28:36Z
Scheduled    2026-09-23T19:28:39Z
PreparingTarget    2026-09-23T19:28:39Z
TargetReady    2026-09-23T19:28:39Z
Running    2026-09-23T19:28:39Z
Succeeded    2026-09-23T19:28:41Z

Succeeded is at 19:28:41. The lost probes are at 1790191721, which is also 19:28:41 UTC. The loss falls in the second the migration finished.

I can't split those ~600 ms. Part of it is the VM paused at the end of the migration. Part of it is the new Type-2 route reaching leaf2.

The sequence number

After the first move, from cp1 to cp2, spine1 read:

[root@spine1 ~]# vtysh -c 'show bgp l2vpn evpn route type macip' | grep -i  -A3 0a:58:c0:a8:28:20
 *>i[2]:[0]:[48]:[0a:58:c0:a8:28:20]
                    192.168.31.22                 100      0 i
                    RT:65000:100 ET:8 MM:1
 *>i[2]:[0]:[48]:[0a:58:c0:a8:28:20]:[32]:[192.168.40.32]
                    192.168.31.22                 100      0 i
                    RT:65000:100 ET:8 MM:1

and leaf2:

leaf2# show evpn mac vni 100 mac 0a:58:c0:a8:28:20
MAC: 0a:58:c0:a8:28:20
 Remote VTEP: 192.168.31.22
 Sync-info: neigh#: 0
 Local Seq: 0 Remote Seq: 1
 Uptime: 00:22:57
 Neighbors:
    192.168.40.32 Active

The MAC is the same, the next hop is now cp2, and MM:1 is the MAC Mobility community with sequence 1. The cluster announced a move rather than a new MAC. FRR on leaf2 accepted it, and re-pointed its forwarding entry from BGP.

After the move back to cp1:

[root@spine1 ~]#  vtysh -c 'show bgp l2vpn evpn route type macip' | grep -i -A3 0a:58:c0:a8:28:20
 *>i[2]:[0]:[48]:[0a:58:c0:a8:28:20]
                    192.168.31.21                 100      0 i
                    RT:65000:100 ET:8 MM:2
 *>i[2]:[0]:[48]:[0a:58:c0:a8:28:20]:[32]:[192.168.40.32]
                    192.168.31.21                 100      0 i
                    RT:65000:100 ET:8 MM:2

MM:2. The VM went back to a VTEP it had used before.

The upstream OVN-Kubernetes EVPN documentation describes this for KubeVirt live migration: "the target node advertises the same MAC/IP with a higher sequence number." In my lab it did, both times.

The VM IP address

Before the move:

❯ oc get vmi -n vm-a -o wide
NAME    AGE   PHASE     IP              NODENAME       READY   LIVE-MIGRATABLE   PAUSED
vm-a1   10m   Running   192.168.40.32   ocp-evpn-cp1   True    True

After it:

❯ oc get vmi -n vm-a -o wide
NAME    AGE   PHASE     IP              NODENAME       READY   LIVE-MIGRATABLE   PAUSED
vm-a1   29m   Running   192.168.40.32   ocp-evpn-cp2   True    True

The node changed, but the IP address did not. The IPAMClaim still holds 192.168.40.32/24. Persistent did exactly what its name suggests: the VM kept its IP address across a live migration.

One detail I noticed in this setup: because I reused the fabric's existing gateway address (192.168.40.1) as the CUDN gateway, both the fabric and OVN-Kubernetes answered ARP for the same IP with different MAC addresses. This is specific to how I built this lab and is separate from the L2 VM migration use case.

What the subnet costs other networks

This is one thing I can take away from this test.

The pods from the last post are in tenant-a on OpenShift, and they route to host-a over VNI 101. OpenShift isolates user-defined networks from each other by design. The documentation says they "are isolated by default."

cudn-vm-a covers 192.168.40.0/24, and host-a lives at .10, inside it. To OVN-Kubernetes, a tenant-a pod sending to .10 looks like traffic crossing into another user-defined network. So it is dropped.

I tested this one change at a time, with no VM. With only the vm-a namespace created, a tenant-a pod to host-a:

== tenant-pod -> 192.168.40.10
64 bytes from 192.168.40.10: icmp_seq=1 ttl=62 time=2.46 ms
64 bytes from 192.168.40.10: icmp_seq=2 ttl=62 time=1.60 ms
64 bytes from 192.168.40.10: icmp_seq=3 ttl=62 time=1.23 ms
3 packets transmitted, 3 received, 0% packet loss, time 2002ms

After creating cudn-vm-a, before even adding its RouteAdvertisements:

== tenant-pod -> 192.168.40.10
3 packets transmitted, 0 received, 100% packet loss, time 2041ms

The lesson is that putting an EVPN network on a subnet already used by another network has a cost.
In my lab, 192.168.40.0/24 is both the fabric-facing subnet for the VM network and the subnet where host-a lives.

That means this post's setup and the last post's setup cannot coexist as-is. The addressing needs to be planned before the first oc apply. But this is exactly why I test these things in the lab. I found a constraint I would not have seen from the configuration alone.

Conclusion

A VM on OpenShift joined a subnet the fabric already stretched. It then moved between nodes and kept its IP and MAC. The fabric treated the move as an EVPN MAC move. The same Type-2 route came back from a new VTEP with a higher sequence number, MM:1 and then MM:2, and leaf2 re-pointed one forwarding entry. host-b lost two pings.

The 616 ms is how this lab behaved, not a production number. Every node here, including the leaves, spine and cluster nodes, is a VM on one Proxmox host. There was one VM, it was idle during both migrations, and the pings were 0.2 s apart. A guest writing memory quickly would take longer to migrate, and I didn't try one.

The migration worked, but I found a planning mistake. Putting the VM network on the fabric’s own subnet cut the tenant-a pods off from host-a. So the address plan has to come first.

What's next

So far, these posts have carried cluster traffic across the fabric inside VXLAN tunnels.

Next I plan to try NoOverlayMode with CUDN, where pod traffic is routed on the underlay without a tunnel.

Thank you for reading.

See you!

NoOverlay mode on OpenShift 4.22

Marvin — Thu, 29 Oct 2026 21:14:45 +0000

Cross-posted from otherthings.cloud, where the rest of my homelab notes live.

The last post ended by saying NoOverlay mode was next. This is that post.

In OpenShift 4.22, a user-defined network can drop the overlay entirely. Pod traffic between nodes goes out as plain IP. Each node's pod subnet is advertised over BGP, and the network routes it like any other prefix.

The documentation gives one reason for it:

You can use no-overlay mode to route layer 3 pod traffic directly over the underlay network with BGP, which reduces encapsulation overhead and improves east-west performance.

Performance is the one thing I can't measure honestly. My cluster is three nested VMs on one Proxmox host. So this post leaves speed for later and asks the other questions: what goes on the wire, what the network sees, and what it took to get there.

TL;DR

With BGP on the node's primary NIC, a pod reaches the outside world with its own IP, and the outside world reaches it back, without any change to host settings. My first build used a second NIC for BGP, and that version didn't round-trip. And two NoOverlay networks stayed isolated from each other, even though the fabric would happily have routed between them.

I hope the diagram is readable at this size. If it isn't, click it to open it full size.

This is the final layout, with BGP on the primary NIC. VLAN 30 is idle here, but my first build peered over it, as the next sections show. The packets in "On the wire" were captured during that first build.

The setup: one cluster, both paths

No-overlay mode is Technology Preview in 4.22. Using it requires the TechPreviewNoUpgrade feature set, which can't be undone and blocks minor-version updates. I didn't want that on the cluster from the earlier posts, so I built a separate three-node cluster, ocp-nooverlay, for this one.

The default pod network stayed on Geneve. Only a new CUDN uses NoOverlay. That gives one cluster with both paths side by side, and every other variable is the same.

The docs show a noOverlayOptions block on the CUDN. On my build, with the feature gate on, that field doesn't exist:

error: field "noOverlayOptions" does not exist

So the CUDN carries transport: NoOverlay and nothing else. Its first status was not what I expected:

TransportAccepted    False    No RouteAdvertisements CR is advertising the pod networks.

The transport isn't accepted until something advertises the network. Once the RouteAdvertisements existed, it changed:

TransportAccepted   True    Transport has been configured as 'no-overlay'.

First build: BGP on a second NIC

Each node has two NICs. The first is the machine network (VLAN 20, behind br-ex). The second sits on VLAN 30, where spine1, the route reflector from the earlier posts, lives. The EVPN cluster peered with spine1 over that second NIC, so I copied the layout.

The routes arrived as planned. Each node learned the other two nodes' pod /24s from spine1, with next hops on VLAN 30:

== ocp-nooverlay-cp1
192.168.52.0/24 nhid 378 via 192.168.30.50 dev enp6s19 proto bgp metric 20
192.168.53.0/24 nhid 376 via 192.168.30.51 dev enp6s19 proto bgp metric 20

Pod to pod across nodes worked:

64 bytes from 192.168.52.5: icmp_seq=1 ttl=60 time=20.6 ms

On the wire

I expected that ping on the VLAN 30 NIC, enp6s19, since that's where the routes point. The capture there caught nothing: a 24-byte file, just the pcap header.

The packets were on the other NIC. A capture on all interfaces found them leaving through enp6s18, the machine network:

22:35:49.274114 enp6s18 Out ifindex 2 52:54:00:00:20:51 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 62, id 4202, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.54.5 > 192.168.52.5: ICMP echo request, id 9, seq 1, length 64

It's plain ICMP, pod IP to pod IP, with no outer header. The IP packet is 84 bytes.

For comparison, here is a ping between two pods on the default Geneve network, on the same NIC:

22:33:10.419329 enp6s18 Out ifindex 2 52:54:00:00:20:51 ethertype IPv4 (0x0800), length 162: (tos 0x0, ttl 64, id 13416, offset 0, flags [DF], proto UDP (17), length 142)
    192.168.20.51.63217 > 192.168.20.52.6081: Geneve, Flags [C], vni 0xff0003, proto TEB (0x6558), options [class Open Virtual Networking (OVN) (0x102) type 0x80(C) len 8]
    0a:58:64:58:00:04 > 0a:58:64:58:00:03, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 26285, offset 0, flags [DF], proto ICMP (1), length 84)
    10.130.0.146 > 10.128.0.173: ICMP echo request, id 8, seq 1, length 64

The same 84-byte ICMP packet travels inside a 142-byte UDP packet to port 6081. That's 58 bytes of encapsulation: the 50 bytes VXLAN costs, plus an 8-byte OVN option. NoOverlay adds nothing.

The MTU didn't change

With no encapsulation, I expected the pod MTU to grow to the underlay's 1500. It didn't:

3: ovn-udn1@if157: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP mode DEFAULT group default qlen 1000

A 1500-byte ping with don't-fragment set never left the pod:

ping: local error: message too long, mtu=1400

The NoOverlay CUDN took the cluster's 1400. The CUDN spec has an mtu field, but I haven't tested whether a NoOverlay network accepts 1500 there.

Off the cluster: out un-NATed, nothing back

The next question was what the network sees when a pod talks to something outside the cluster. I pinged spine1 from a NoOverlay pod and captured on spine1:

18:12:24.444219 48:a9:8a:38:16:62 > bc:24:11:3a:b2:ba, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 61, id 32720, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.54.5 > 192.168.30.2: ICMP echo request, id 12, seq 1, length 64
18:12:24.444624 bc:24:11:3a:b2:ba > 52:54:00:00:30:49, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 65249, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.30.2 > 192.168.54.5: ICMP echo reply, id 12, seq 1, length 64

The source is 192.168.54.5, the pod's own address, not the node's. That's what I wanted to see.

spine1 replied, too. But the pod never got the reply:

5 packets transmitted, 0 received, 100% packet loss, time 4072ms

Pinging the pod from spine1 failed the same way.

The two directions took different paths. The request left the node by its default route, via 192.168.20.1 dev br-ex, on the machine network, and was routed to spine1 from there. The reply followed spine1's BGP route, which points at the node's VLAN 30 address. So the reply arrived on enp6s19.

OVN-Kubernetes doesn't manage enp6s19, and forwarding is off on it:

net.ipv4.conf.enp6s19.forwarding = 0

The node knows where the pod is. It just won't forward a packet arriving on that NIC:

[root@ocp-nooverlay-cp1 /]# ip route get 192.168.54.5 from 192.168.30.2 iif enp6s19
RTNETLINK answers: No route to host

Host settings on that NIC can change this. I didn't want a design that depends on tuning the host. So I moved the BGP session instead.

Peering on the primary NIC

I gave spine1 a leg on VLAN 20 and pointed the cluster's FRRConfiguration at that address instead. Nothing else in the cluster changed.

FRR-K8s used each node's VLAN 20 address as the next hop with no extra configuration:

 *>i192.168.52.0/24  192.168.20.52            0    100      0 i
 *>i192.168.53.0/24  192.168.20.53            0    100      0 i
 *>i192.168.54.0/24  192.168.20.51            0    100      0 i

Then the same test again, from both ends, captured on spine1's VLAN 20 NIC, ens19.

The pod pings spine1:

13:32:11.370234 ens19 In  ifindex 3 52:54:00:00:20:51 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 62, id 53389, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.54.5 > 192.168.20.25: ICMP echo request, id 44, seq 1, length 64
13:32:11.370396 ens19 Out ifindex 3 52:54:00:00:20:25 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 64, id 22216, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.20.25 > 192.168.54.5: ICMP echo reply, id 44, seq 1, length 64

spine1 pings the pod at its own address:

13:32:34.108343 ens19 Out ifindex 3 52:54:00:00:20:25 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 64, id 25468, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.20.25 > 192.168.54.5: ICMP echo request, id 4, seq 1, length 64
13:32:34.110169 ens19 In  ifindex 3 52:54:00:00:20:51 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 62, id 7458, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.54.5 > 192.168.20.25: ICMP echo reply, id 4, seq 1, length 64

Both directions were 5/5. There's no NodePort, Service or LoadBalancer in the path. The pod is just an address on the network.

A pod on the default Geneve network, pinging the same target over the same wire, arrives as its node:

13:32:38.383135 ens19 In  ifindex 3 52:54:00:00:20:51 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 62, id 32935, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.20.51 > 192.168.20.25: ICMP echo request, id 48889, seq 1, length 64

That's the whole difference between the two networks, as spine1 sees it: 192.168.54.5 against 192.168.20.51.

Afterwards I read the node's settings again. Nothing had been changed:

net.ipv4.ip_forward = 0
net.ipv4.conf.enp6s19.forwarding = 0
net.ipv4.conf.enp6s19.rp_filter = 1
net.ipv4.conf.enp6s18.forwarding = 0
net.ipv4.conf.br-ex.forwarding = 1

Forwarding is on only for br-ex, the interface OVN-Kubernetes manages. With BGP on the primary NIC, traffic arrives on br-ex, so that's all it needs.

Isolation still holds

One question was still bothering me. With Geneve, two user-defined networks are kept apart, and I'd measured that in an earlier lab: pods on different UDNs can't reach each other. But with NoOverlay there is no tunnel to keep them apart. Every network is just routes.

So I created a second NoOverlay CUDN, B, next to A. spine1 now holds both networks' subnets in one table:

 *>i192.168.52.0/24  192.168.20.52            0    100      0 i
 *>i192.168.53.0/24  192.168.20.53            0    100      0 i
 *>i192.168.54.0/24  192.168.20.51            0    100      0 i
 *>i192.168.56.0/24  192.168.20.52            0    100      0 i
 *>i192.168.57.0/24  192.168.20.53            0    100      0 i
 *>i192.168.58.0/24  192.168.20.51            0    100      0 i

The node had a route too. On cp1, the kernel's table for network A knows how to reach B's pod on cp2:

192.168.56.4 from 192.168.54.5 via 192.168.20.52 dev br-ex table 1067

Both controls passed first: B's pods reached each other, and spine1 reached B's pod. Then A to B:

PING 192.168.56.4 (192.168.56.4) 56(84) bytes of data.

--- 192.168.56.4 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4078ms

A to B failed on the same node and across nodes, and B to A failed too. Every case lost every packet, silently. That's the same result, with the same silence, as Geneve UDNs.

The kernel route was never used. In a capture on cp1, the packets appear only on pod A's own port, and never on enp6s18. The capture on cp2 was empty:

14:41:05.545535 ?     P   ifindex 157 0a:58:c0:a8:36:05 ethertype IPv4 (0x0800), length 104: (tos 0x0, ttl 64, id 16698, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.54.5 > 192.168.56.4: ICMP echo request, id 52, seq 1, length 64

So the packet is dropped inside the first node, before it reaches any routing table.

The rule that drops it

NOTE: The commands for these traces and lookups, so you can find the same rules on your own cluster, are in isolation-trace.md.

To find out where, I used ovn-trace. OVN turns its configuration (switches, routers, ACLs) into a long list of numbered rules. ovn-trace runs a made-up packet through that list without sending anything, and prints each rule the packet matches, in order. I gave it a ping from A's pod on cp1 to B's pod.

Most of the rules it passed are routine. This is the one that stopped it:

20. ls_in_acl_after_lb_eval (northd.c:7608): reg8[30..31] == 0 && reg0[9] == 1 && ((ip4.src == $a10109792604843350142 && ip4.dst == $a10109792604843350142)), priority 2050, uuid a5d41540
    reg8[17] = 1;

The part that matters is in the double brackets. If the source and the destination are both in $a10109792604843350142, the rule sets reg8[17], which marks the packet to be dropped. The trace ends at the next rule. Nothing after it forwards the packet.

$a10109792604843350142 is an address set: a named list of subnets that rules can refer to. OVN's database shows what's in it:

addresses           : ["192.168.52.0/22", "192.168.56.0/22"]

Those are the whole subnets of A and B. Each network is a /22, and spine1's table above shows the /24 slice each node takes from it. So the rule drops any packet whose source and destination are both on the list of advertised networks.

That would drop A to A too. It doesn't, because a rule with higher priority wins. When I traced A pod to A pod, this rule matched first and let the packet through:

20. ls_in_acl_after_lb_eval (northd.c:7480): reg8[30..31] == 0 && (((ip4.src == 192.168.52.0/22) && (ip4.dst == 192.168.52.0/22))), priority 2100, uuid 72ab9e68

It matches only traffic that stays inside A's own subnet, and at 2100 it beats the drop rule at 2050. Between them, the two rules allow traffic within a network and drop traffic between networks.

In OVN's database, both the address set and the rule are owned by AdvertisedNetwork, not by anything named for NoOverlay. From that I'd guess the same rule separates any advertised UDNs, Geneve or EVPN included. I haven't tested that.

Conclusion

Peer on the primary NIC, and NoOverlay does what it says: plain packets, the pod's own address on the network, and nothing to tune on the host. Isolation between networks survives the loss of the tunnel, because it never depended on the tunnel.

The part I couldn't measure is the reason the feature exists: performance. That needs real hardware, and it's the next part of this series.

Everything needed to build it is in 2026-10-openshift-nooverlay/.