Static routes work — until they don't. In a multi-VRF NX-OS fabric with an OPNsense border firewall as the WAN edge, a single dead next-hop means a dead service. With an external ISP link (Optus) being decommissioned and the last backup path disappearing with it, I needed something more resilient for the storage VRF — the network segment carrying my DePIN storage node.
This is a phased OSPF migration across three devices — two NX-OS Top-of-Rack switches and an OPNsense border firewall running FRR — executed live, with the storage node serving active sessions the whole time.
// the topology
The storage VRF runs on its own VLAN (illustrative 192.168.x.0/24). Prior to this change, the routing model was purely static:
- TOR-B held a single default static pointing to the border firewall via the TOR-B↔firewall transit (illustrative
/30) - TOR-A held a cross-TOR static pointing to TOR-B
- The border firewall held kernel statics for the storage subnet and the TOR-A↔TOR-B transit, pointing back through TOR-B
One physical path. One point of failure. The Optus link had been providing a last-resort backup — once it went, there would be nothing.
// the fix
The fix was a second physical uplink from TOR-A directly to the border firewall on ix0, tagged on a second transit VLAN (illustrative /30), combined with OSPF process 10 across all three devices. I followed a phased template I'd developed on prior VRF changes:
- P1 — build the TOR-A↔firewall link and verify L3 reachability
- P2 — enable OSPF on all three devices alongside existing statics (observation window)
- P3 — reduce hello/dead timers to 1s/4s for fast failover
- P4 — float the TOR statics to AD 250 (OSPF becomes active path)
- P5 — remove the firewall's kernel statics and save configs (OSPF owns the full routing table)
// P1–P2: building the foundation
P1 was straightforward: NX-OS subinterface on the firewall-facing uplink, OPNsense VLAN interface on ix0, firewall rules permitting the storage VRF's transit prefixes. A ping test confirmed 5/5 at ~1ms before proceeding.
P2 brought OSPF up on all three devices simultaneously — distinct router-ids per device (illustrative 10.x.x.1 on TOR-B, 10.x.x.9 on TOR-A, 10.x.x.6 on the border firewall). The key discipline here: statics were left untouched. OSPF ran in parallel, forming Full adjacencies on both TOR-firewall links and the TOR-A↔TOR-B transit, while all traffic continued to forward via the existing statics. That gave a clean observation window — OSPF converged, LSAs exchanged, default route type-5 LSA from the border firewall visible in both TOR databases — before any forwarding change was made.
// P3: fast timers and a shell gotcha
I applied the timer reduction to 1s hello / 4s dead firewall-first, then both TORs in immediate succession. The sequencing matters: once the firewall drops its dead-interval to 4 seconds, any TOR still sending 10-second hellos will be declared dead within 4 seconds of the next missed hello.
In practice I applied the TOR changes fast enough that no adjacency flap occurred — the Up Time on all neighbours continued uninterrupted through P3. The theoretical risk window was sub-4 seconds; the actual outcome was zero disruption.
One operational note for anyone running FRR on OPNsense (FreeBSD): the vtysh -c '...' multiline quoting form does not work in the FreeBSD shell. Single-quoted multiline strings are parsed by csh/tcsh, which rejects them. The correct pattern is chained -c flags on a single line:
vtysh -c 'configure terminal' -c 'interface vlanXX' -c 'ip ospf hello-interval 1' -c 'ip ospf dead-interval 4' -c 'end' -c 'write memory'
// P4–P5: the forwarding cutover
Floating the statics to AD 250 at P4 was the actual forwarding cutover. With OSPF already converged at AD 110, removing the AD 1 statics caused both TORs to immediately prefer the OSPF-learned default. The cutover was effectively instantaneous — OSPF had been sitting ready for the duration of the P2 observation window.
To be precise about the "zero-blip" characterisation: the forwarding plane change at P4 was zero-blip. OSPF was already converged; the static removal was atomic. P3 carried a theoretical sub-4s adjacency risk that did not materialise. The claim holds for the forwarding plane; it should be understood as "no measured disruption" rather than a mathematically guaranteed zero.
At P5 I removed the firewall's kernel statics for the storage subnet and the second transit, and saved configs. Post-removal, both prefixes were confirmed OSPF-selected with ECMP across both uplinks — exactly the redundancy the whole exercise was designed to achieve.
// the result
The storage VRF now has:
- Dual OSPF paths to the border firewall — ECMP load-sharing under normal conditions
- Automatic failover on either TOR-firewall link loss — convergence up to 4 seconds by design (1s hello, 4s dead-interval); a measured figure will be captured in a controlled failover test
- Floating statics at AD 250 as a silent safety net if OSPF fails entirely
- The storage node serving active sessions throughout — zero session interruptions recorded in the storage node's own session log across the migration window
// what I'd make instinctive
Four things from this.
Run OSPF alongside statics before cutting over. The P2 observation window — OSPF converged but statics still forwarding — is not optional. It confirms adjacencies, LSA propagation, and the default route type-5 before any forwarding change is made. Skipping it trades safety for speed and gains nothing.
NX-OS vrf member silently removes all L3 config on an SVI. Adding vrf member to an existing SVI in NX-OS drops every IP address and routing statement on that interface without warning. Always build subinterfaces or SVIs with the VRF assignment as the first command, before any L3 config.
Timer sequencing is firewall-first, TORs immediately after. Apply fast timers to the device with the shorter dead-interval first — in this case the border firewall. Then apply to both TORs as quickly as possible. The window between firewall update and TOR update is a live risk interval; keep it under 4 seconds.
The floating static is not dead weight. AD 250 statics provide a recovery path if the OSPF process fails for any reason — misconfiguration, FRR crash, process restart. They cost nothing in normal operation and provide a last-resort forwarding path. Keep them.
Originally published at jjozzietech.com.au
Top comments (0)