Cross-posted from otherthings.cloud, where the rest of my homelab notes live.
The last post joined the OpenShift 4.22 cluster to the fabric through the IP-VRF: pods on their own subnet, routed to host-a. It ended with the MAC-VRF side untested. This post tests that side.
A MAC-VRF stretches a Layer 2 segment across the EVPN fabric for east-west bridged traffic and workloads such as VM live migration.
NOTE: See the OVN-Kubernetes EVPN documentation for the MAC-VRF definition.
I wanted a VM on OpenShift to sit on the same subnet as host-a and host-b: 192.168.40.0/24. The two leaves already stretch this subnet between them as VNI 100 in my lab.
Then I wanted to live-migrate the VM to another node (cp1 -> cp2) while host-b continuously pinged it, and see what the fabric made of the move.
The VM kept its IP and its MAC. The fabric treated the migration as an EVPN MAC move, with a sequence number increasing each time. host-b lost two pings during the migration.
There was also a subnet-planning consequence that I found later in the test.
spine1 192.168.30.2
(BGP route reflector)
┌───────────────┬────────┬────────┬───────────┐
│ │ │ │ │
leaf1 leaf2 cp1 cp2 cp3
192.168.30.11 .30.12 .31.21 .31.22 .31.23
VTEP VTEP VTEP VTEP VTEP
│ │ └────────┬──────-─────┘
host-a host-b │
.40.10 .40.11 vm-a1
│ │ .40.32
└──────────────┴────────────────┘
192.168.40.0/24
VNI 100
cp1 ─────→ cp2
VM migration
cluster nodes as VTEPs: cp1 192.168.31.21 · cp2 .22 · cp3 .23
vm-a1 192.168.40.32 — moves between cp1 and cp2
vm-a1 starts on worker cp1 and is live-migrated to worker cp2, while host-b continuously pings vm-a1:192.168.40.32.
The network
The VM's network is a ClusterUserDefinedNetwork with topology: Layer2 and transport: EVPN. In other words, this creates a Layer 2 primary CUDN across the EVPN network as MAC-VRF:
evpn:
vtep: evpn-vtep
macVRF:
vni: 100
routeTarget: "65000:100"
layer2:
role: Primary
subnets: ["192.168.40.0/24"]
infrastructureSubnets: ["192.168.40.0/29"]
defaultGatewayIPs: ["192.168.40.1"]
reservedSubnets:
- "192.168.40.8/29"
- "192.168.40.16/28"
ipam:
mode: Enabled
lifecycle: Persistent
mtu: 1400
Three parts of it matter here.
macVRF.vni: 100 is the number the leaves already use for this segment. The VNI is what identifies an L2 segment across the fabric. Using the same number puts the VM in the same broadcast domain as host-a and host-b.
The subnet lists are a fence. By default OVN-Kubernetes allocates from the whole /24, but .1, .10 and .11 already belong to the fabric. infrastructureSubnets holds the gateway, and reservedSubnets keeps .8 to .31 away from the allocator. VMs get .32 and up. The spec is immutable, so the fence cannot be widened later.
ipam.lifecycle: Persistent is what should keep the VM's address across a migration. Whether it does is one of the things I checked.
There is no ipVRF. My first design had one, using VNI 101, the fabric's tenant VRF. The API accepted the object, and then this appeared in its NetworkCreated condition:
"message": "failed to allocate EVPN IDs: failed to reserve VNIs: VNI 101 on VTEP \"evpn-vtep\" is already reserved by network \"cudn-tenant-a\"",
cudn-tenant-a is the network from the last post, and it already holds 101. A VNI can be reserved by only one network on a VTEP. So this network is L2 only.
For this test, that costs nothing. The VM reaches host-a and host-b by bridging, and bridging needs no VRF. It does mean the VM's segment has no routed path to anything off the subnet.
The VM runs CentOS Stream 9 with the l2bridge interface binding. Its disk is an RWX PVC, because live migration needs shared storage. Before anything moved, the VMI already reported LIVE-MIGRATABLE True.
Everything needed to build it — the four manifests in apply order, and the commands for watching a migration — is in other-things-lab/2026-09-openshift-virt-macvrf.
Bridged, not routed
From the VM's console, to host-b:
[cloud-user@vm-a1 ~]$ ping -c 5 192.168.40.11
PING 192.168.40.11 (192.168.40.11) 56(84) bytes of data.
64 bytes from 192.168.40.11: icmp_seq=1 ttl=64 time=4.39 ms
64 bytes from 192.168.40.11: icmp_seq=2 ttl=64 time=1.58 ms
64 bytes from 192.168.40.11: icmp_seq=3 ttl=64 time=1.19 ms
64 bytes from 192.168.40.11: icmp_seq=4 ttl=64 time=1.25 ms
64 bytes from 192.168.40.11: icmp_seq=5 ttl=64 time=1.29 ms
--- 192.168.40.11 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
rtt min/avg/max/mdev = 1.194/1.939/4.387/1.230 ms
ttl=64 means no router is in the path. Linux sends with 64, and every router hop takes one off.
On leaf1, during the same test to host-a:
[root@leaf1 ~]# tcpdump -ni ens18 'udp port 4789' -c 10 -v
dropped privs to tcpdump
tcpdump: listening on ens18, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:31:59.271470 IP (tos 0x0, ttl 64, id 61870, offset 0, flags [none], proto UDP (17), length 134)
192.168.31.21.57978 > 192.168.30.11.vxlan: VXLAN, flags [I] (0x08), vni 100
IP (tos 0x0, ttl 64, id 50680, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.40.32 > 192.168.40.10: ICMP echo request, id 2, seq 1, length 64
13:31:59.272963 IP (tos 0x0, ttl 64, id 17652, offset 0, flags [none], proto UDP (17), length 134)
192.168.30.11.44658 > 192.168.31.21.vxlan: VXLAN, flags [I] (0x08), vni 100
IP (tos 0x0, ttl 64, id 54707, offset 0, flags [none], proto ICMP (1), length 84)
192.168.40.10 > 192.168.40.32: ICMP echo reply, id 2, seq 1, length 64
The request arrives in VNI 100, from 192.168.31.21, which is cp1's VTEP. The reply goes back the same way. The VM is on the fabric's segment itself, not on a network routed to it.
Before the VM migration
The VM's MAC is 0a:58:c0:a8:28:20. The last four bytes are 192.168.40.32 in hex, because OVN-Kubernetes derives the MAC from the IP. So if the IP survives the migration, the MAC should survive too.
On spine1, the VM has two Type-2 routes, one for the MAC alone and one for MAC plus IP:
[root@spine1 ~]# vtysh -c 'show bgp l2vpn evpn route type macip' | grep -i -A3 '0a:58:c0:a8:28:20'
*>i[2]:[0]:[48]:[0a:58:c0:a8:28:20]
192.168.31.21 100 0 i
RT:65000:100 ET:8
*>i[2]:[0]:[48]:[0a:58:c0:a8:28:20]:[32]:[192.168.40.32]
192.168.31.21 100 0 i
RT:65000:100 ET:8
Both have next hop 192.168.31.21, which is cp1.
On leaf2, which is where host-b's traffic enters the fabric:
[root@leaf2 ~]# vtysh -c 'show evpn mac vni 100 mac 0a:58:c0:a8:28:20'
MAC: 0a:58:c0:a8:28:20
Remote VTEP: 192.168.31.21
Sync-info: neigh#: 0
Local Seq: 0 Remote Seq: 0
Uptime: 00:14:20
Neighbors:
192.168.40.32 Active
The Remote Seq: 0 is the MAC Mobility sequence number. RFC 7432 describes this sequence number as part of the mechanism used to detect MAC moves. When a MAC moves, the new advertisement can carry a higher sequence number, allowing other VTEPs to prefer the newer location.
The alternative would be for the cluster to withdraw the old route and advertise a fresh one. In that case the sequence number would stay at 0.
The kernel forwarding entry on leaf2 is also worth a look:
[root@leaf2 ~]# bridge fdb show | grep -i '0a:58:c0:a8:28:20'
0a:58:c0:a8:28:20 dev vxlan100 vlan 1 extern_learn master br100
0a:58:c0:a8:28:20 dev vxlan100 extern_learn master br100
0a:58:c0:a8:28:20 dev vxlan100 dst 192.168.31.21 self extern_learn
extern_learn means FRR installed the entry from BGP. The leaf did not learn the VM's MAC from traffic.
The important point before migration is therefore:
leaf2 currently knows that MAC 0a:58:c0:a8:28:20 is reachable through VTEP 192.168.31.21 (cp1).
The VM live migration
I migrated the VM in both directions: first from cp1 to cp2, and then back from cp2 to cp1. I'm quoting the second run, because its ping output is complete text and leaf2 wrote its capture to a file. The first run behaved the same way, but its ping summary survived only as a screenshot.
host-b ran ping -D -O -i 0.2 192.168.40.32. -D adds a timestamp to each line, and -O prints a line for every probe that gets no reply:
[1790191720.254543] 64 bytes from 192.168.40.32: icmp_seq=296 ttl=64 time=2.39 ms
[1790191720.454958] 64 bytes from 192.168.40.32: icmp_seq=297 ttl=64 time=1.99 ms
[1790191720.654829] 64 bytes from 192.168.40.32: icmp_seq=298 ttl=64 time=1.41 ms
[1790191721.060312] no answer yet for icmp_seq=299 ← migration
[1790191721.268299] no answer yet for icmp_seq=300
[1790191721.273476] 64 bytes from 192.168.40.32: icmp_seq=301 ttl=64 time=2.24 ms
[1790191721.470693] 64 bytes from 192.168.40.32: icmp_seq=302 ttl=64 time=1.77 ms
The two missing replies (icmp_seq=299 and 300) are the only packet-loss window during the migration.
From host-b's side, 0.619 s passed between the reply to 298 and the reply to 301.
leaf2 saw the same probes. Its clock is on local time, five hours behind the UTC timestamps above:
14:28:40.600748 IP 192.168.30.12.44524 > 192.168.31.22.4789: VXLAN, flags [I] (0x08), vni 100 IP 192.168.40.11 > 192.168.40.32: ICMP echo request, id 5, seq 298, length 64
14:28:40.601534 IP 192.168.31.22.53542 > 192.168.30.12.4789: VXLAN, flags [I] (0x08), vni 100 IP 192.168.40.32 > 192.168.40.11: ICMP echo reply, id 5, seq 298, length 64
14:28:40.801638 IP 192.168.30.12.44524 > 192.168.31.22.4789: VXLAN, flags [I] (0x08), vni 100 IP 192.168.40.11 > 192.168.40.32: ICMP echo request, id 5, seq 299, length 64
14:28:41.007759 IP 192.168.30.12.44524 > 192.168.31.22.4789: VXLAN, flags [I] (0x08), vni 100 IP 192.168.40.11 > 192.168.40.32: ICMP echo request, id 5, seq 300, length 64
14:28:41.215730 IP 192.168.30.12.44524 > 192.168.31.21.4789: VXLAN, flags [I] (0x08), vni 100 IP 192.168.40.11 > 192.168.40.32: ICMP echo request, id 5, seq 301, length 64
14:28:41.217416 IP 192.168.31.21.49680 > 192.168.30.12.4789: VXLAN, flags [I] (0x08), vni 100 IP 192.168.40.32 > 192.168.40.11: ICMP echo reply, id 5, seq 301, length 64
Seq 298 went to 192.168.31.22, cp2, and was answered.
Seq 299 and 300 also went to cp2, but the VM was no longer answering there.
Seq 301 went to 192.168.31.21, cp1, and was answered 1.7 ms later.
So the fabric didn't drop the lost probes. It delivered them to where the VM had been. Between 41.007759 and 41.215730, leaf2 changed one destination. Nothing was flooded, and no packet went to both nodes.
As leaf2 saw it, 616 ms passed between the last reply from cp2 and the first reply from cp1. The VM was unreachable for somewhere between about 0.2 s and 0.6 s.
The migration's own timeline, in UTC:
❯ oc get vmim kubevirt-migrate-vm-bh22k -n vm-a -o jsonpath='{range.status.phaseTransitionTimestamps[*]}{.phase}{"\t"}{.phaseTransitionTimestamp}{"\n"}{end}'
Pending 2026-09-23T19:28:36Z
Scheduling 2026-09-23T19:28:36Z
Scheduled 2026-09-23T19:28:39Z
PreparingTarget 2026-09-23T19:28:39Z
TargetReady 2026-09-23T19:28:39Z
Running 2026-09-23T19:28:39Z
Succeeded 2026-09-23T19:28:41Z
Succeeded is at 19:28:41. The lost probes are at 1790191721, which is also 19:28:41 UTC. The loss falls in the second the migration finished.
I can't split those ~600 ms. Part of it is the VM paused at the end of the migration. Part of it is the new Type-2 route reaching leaf2.
The sequence number
After the first move, from cp1 to cp2, spine1 read:
[root@spine1 ~]# vtysh -c 'show bgp l2vpn evpn route type macip' | grep -i -A3 0a:58:c0:a8:28:20
*>i[2]:[0]:[48]:[0a:58:c0:a8:28:20]
192.168.31.22 100 0 i
RT:65000:100 ET:8 MM:1
*>i[2]:[0]:[48]:[0a:58:c0:a8:28:20]:[32]:[192.168.40.32]
192.168.31.22 100 0 i
RT:65000:100 ET:8 MM:1
and leaf2:
leaf2# show evpn mac vni 100 mac 0a:58:c0:a8:28:20
MAC: 0a:58:c0:a8:28:20
Remote VTEP: 192.168.31.22
Sync-info: neigh#: 0
Local Seq: 0 Remote Seq: 1
Uptime: 00:22:57
Neighbors:
192.168.40.32 Active
The MAC is the same, the next hop is now cp2, and MM:1 is the MAC Mobility community with sequence 1. The cluster announced a move rather than a new MAC. FRR on leaf2 accepted it, and re-pointed its forwarding entry from BGP.
After the move back to cp1:
[root@spine1 ~]# vtysh -c 'show bgp l2vpn evpn route type macip' | grep -i -A3 0a:58:c0:a8:28:20
*>i[2]:[0]:[48]:[0a:58:c0:a8:28:20]
192.168.31.21 100 0 i
RT:65000:100 ET:8 MM:2
*>i[2]:[0]:[48]:[0a:58:c0:a8:28:20]:[32]:[192.168.40.32]
192.168.31.21 100 0 i
RT:65000:100 ET:8 MM:2
MM:2. The VM went back to a VTEP it had used before.
The upstream OVN-Kubernetes EVPN documentation describes this for KubeVirt live migration: "the target node advertises the same MAC/IP with a higher sequence number." In my lab it did, both times.
The VM IP address
Before the move:
❯ oc get vmi -n vm-a -o wide
NAME AGE PHASE IP NODENAME READY LIVE-MIGRATABLE PAUSED
vm-a1 10m Running 192.168.40.32 ocp-evpn-cp1 True True
After it:
❯ oc get vmi -n vm-a -o wide
NAME AGE PHASE IP NODENAME READY LIVE-MIGRATABLE PAUSED
vm-a1 29m Running 192.168.40.32 ocp-evpn-cp2 True True
The node changed, but the IP address did not. The IPAMClaim still holds 192.168.40.32/24. Persistent did exactly what its name suggests: the VM kept its IP address across a live migration.
One detail I noticed in this setup: because I reused the fabric's existing gateway address (192.168.40.1) as the CUDN gateway, both the fabric and OVN-Kubernetes answered ARP for the same IP with different MAC addresses. This is specific to how I built this lab and is separate from the L2 VM migration use case.
What the subnet costs other networks
This is one thing I can take away from this test.
The pods from the last post are in tenant-a on OpenShift, and they route to host-a over VNI 101. OpenShift isolates user-defined networks from each other by design. The documentation says they "are isolated by default."
cudn-vm-a covers 192.168.40.0/24, and host-a lives at .10, inside it. To OVN-Kubernetes, a tenant-a pod sending to .10 looks like traffic crossing into another user-defined network. So it is dropped.
I tested this one change at a time, with no VM. With only the vm-a namespace created, a tenant-a pod to host-a:
== tenant-pod -> 192.168.40.10
64 bytes from 192.168.40.10: icmp_seq=1 ttl=62 time=2.46 ms
64 bytes from 192.168.40.10: icmp_seq=2 ttl=62 time=1.60 ms
64 bytes from 192.168.40.10: icmp_seq=3 ttl=62 time=1.23 ms
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
After creating cudn-vm-a, before even adding its RouteAdvertisements:
== tenant-pod -> 192.168.40.10
3 packets transmitted, 0 received, 100% packet loss, time 2041ms
The lesson is that putting an EVPN network on a subnet already used by another network has a cost.
In my lab, 192.168.40.0/24 is both the fabric-facing subnet for the VM network and the subnet where host-a lives.
That means this post's setup and the last post's setup cannot coexist as-is. The addressing needs to be planned before the first oc apply. But this is exactly why I test these things in the lab. I found a constraint I would not have seen from the configuration alone.
Conclusion
A VM on OpenShift joined a subnet the fabric already stretched. It then moved between nodes and kept its IP and MAC. The fabric treated the move as an EVPN MAC move. The same Type-2 route came back from a new VTEP with a higher sequence number, MM:1 and then MM:2, and leaf2 re-pointed one forwarding entry. host-b lost two pings.
The 616 ms is how this lab behaved, not a production number. Every node here, including the leaves, spine and cluster nodes, is a VM on one Proxmox host. There was one VM, it was idle during both migrations, and the pings were 0.2 s apart. A guest writing memory quickly would take longer to migrate, and I didn't try one.
The migration worked, but I found a planning mistake. Putting the VM network on the fabric’s own subnet cut the tenant-a pods off from host-a. So the address plan has to come first.
What's next
So far, these posts have carried cluster traffic across the fabric inside VXLAN tunnels.
Next I plan to try NoOverlayMode with CUDN, where pod traffic is routed on the underlay without a tunnel.
Thank you for reading.
See you!
Top comments (0)