Spine and Leaf Practical Applications, EGP and IGP combined!

So far, all examples to date have been extremely simple, and stand rather well on their own (OSPF Clos would work just FINE in a campus network if you have cheap L3) but may not effectively address some more advanced use cases.

In short, we're about to enter niche territory.

Generally speaking, BGP is regarded as the most advanced networking topic out there, until it isn't. Most of the complexity lies in iBGP (same Autonomous System everywhere), because BGP's primary loop prevention mechanism is AS-Path (count on the number of AS this route transits).

To successfully implement iBGP (a staple of every carrier, so totally do-able), a network engineer must choose one of the following paths (non-inclusive list):

• Fully mesh all BGP speakers with each other (factorial scaling)
• Implement a Route Reflector: https://en.wikipedia.org/wiki/Route_reflector

In this case, the more scalable option is to opt for a route reflector, but it isn't that easy.

iBGP, when compared to eBGP:

• Doesn't care about hop count to a peer speaker. As long as the route reflector is less than 255 hops away, there is no issue
• Usually doesn't care about resolving paths to peer speakers - that's a problem for something else.

To provide a good example of an expandable, scalable fabric that can offer eBGP as a service to subtending network devices, we will implement IS-IS as the intra-fabric routing protocol, and then leverage iBGP with the spine switches as route reflectors.

First things first, diagram is here: (YAML)

First, we'd configure the spines. Note that future releases (my home lab is rockin' IOS 12.2.55, too old for this) the BGP Dynamic Neighbor that comes with more modern network operating systems is really useful.

Note: route reflector client status is configured on the SERVER side:

router bgp 65000 bgp log-neighbor-changes neighbor 10.6.0.0 remote-as 65000 neighbor 10.6.0.0 update-source L0 neighbor 10.6.0.0 route-reflector-client neighbor 10.6.0.1 remote-as 65000 neighbor 10.6.0.1 update-source L0 neighbor 10.6.0.1 route-reflector-client neighbor FD00:6::0 remote-as 65000 neighbor FD00:6::0 update-source L0 neighbor FD00:6::0 route-reflector-client neighbor FD00:6::1 remote-as 65000 neighbor FD00:6::1 update-source L0 neighbor FD00:6::1 route-reflector-client maximum-paths 2 ! address-family ipv4 neighbor 10.6.0.0 activate neighbor 10.6.0.1 activate maximum-paths 2 no auto-summary network 10.6.0.240 mask 255.255.255.254 network 10.6.240.0 mask 255.255.255.254 network 10.6.240.2 mask 255.255.255.254 exit-address-family ! address-family ipv6 neighbor FD00:6::0 activate neighbor FD00:6::1 activate network FD00:6::240/127 network FD00:6:240::/126 network FD00:6:240::4/126 no synchronization maximum-paths 2 exit-address-family!

And then the leaf configuration:

router bgp 65000 bgp log-neighbor-changes neighbor 10.6.0.240 remote-as 65000 neighbor 10.6.0.240 update-source L0 neighbor 10.6.0.241 remote-as 65000 neighbor 10.6.0.241 update-source L0 neighbor FD00:6::240 remote-as 65000 neighbor FD00:6::240 update-source L0 neighbor FD00:6::241 remote-as 65000 neighbor FD00:6::241 update-source L0 maximum-paths 2 ! address-family ipv4 neighbor 10.6.0.240 activate neighbor 10.6.0.241 activate maximum-paths 2 no auto-summary network 10.6.0.240 mask 255.255.255.254 network 10.6.240.0 mask 255.255.255.254 network 10.6.240.2 mask 255.255.255.254 exit-address-family ! address-family ipv6 neighbor FD00:6::240 activate neighbor FD00:6::241 activate network FD00:6::240/127 network FD00:6:240::/126 network FD00:6:240::4/126 no synchronization maximum-paths 2 exit-address-family!

Note that no BGP peers are up yet - and BGP knows what the problem is, too!

bgp-rr0-s0#show ip bgp06:12:15: %SYS-5-CONFIG_I: Configured from console by consoleneighBGP neighbor is 10.6.0.0, remote AS 65000, internal link BGP version 4, remote router ID 0.0.0.0 BGP state = Active Last read 00:03:02, last write 00:03:02, hold time is 180, keepalive interval is 60 seconds Message statistics: InQ depth is 0 OutQ depth is 0 Sent Rcvd Opens: 0 0 Notifications: 0 0 Updates: 0 0 Keepalives: 0 0 Route Refresh: 0 0 Total: 0 0 Default minimum time between advertisement runs is 0 seconds For address family: IPv4 Unicast BGP table version 1, neighbor version 0/0 Output queue size : 0 Index 1, Offset 0, Mask 0x2 Route-Reflector Client 1 update-group member Sent Rcvd Prefix activity: ---- ---- Prefixes Current: 0 0 Prefixes Total: 0 0 Implicit Withdraw: 0 0 Explicit Withdraw: 0 0 Used as bestpath: n/a 0 Used as multipath: n/a 0 Outbound Inbound Local Policy Denied Prefixes: -------- ------- Total: 0 0 Number of NLRIs in the update sent: max 0, min 0 Address tracking is enabled, the RIB does not have a route to 10.6.0.0 Address tracking requires at least a /0 route to the peer Connections established 0; dropped 0 Last reset never Transport(tcp) path-mtu-discovery is enabled No active TCP connection

Note how it says the RIB does not have a route to 10.6.0.0 - that's because iBGP doesn't resolve next-hops for us. Let's fix it by rolling out an Interior Gateway Protocol (IGP) to support iBGP here. I'm using IS-IS for a few reasons - namely:

• Like BGP, one routing protocol for both IPv4 and IPv6
• Selective flooding with ISPF
• I'm too hip for OSPF now

router isis CLOS-1 net 42.0000.0000.0000.0240.00 is-type level-2-only ispf level-2 log-adjacency-changes!interface ip router isis CLOS-1interface Loopback0ip router isis CLOS-1

This is applied to every router, while changing the net-ID for each device.It's fun watching the adjacencies pop up, so I'll add that here too.

*Mar 1 06:31:34.670: %CLNS-5-ADJCHANGE: ISIS: Adjacency to 0000.0000.0240 (FastEthernet0/22) Up, new adjacency*Mar 1 06:31:34.670: %CLNS-5-ADJCHANGE: ISIS: Adjacency to 0000.0000.0241 (FastEthernet0/23) Up, new adjacency *Mar 1 06:31:41.565: %BGP-5-ADJCHANGE: neighbor 10.6.0.240 Up*Mar 1 06:31:47.169: %BGP-5-ADJCHANGE: neighbor 10.6.0.241 Up

Note how BGP pops up immediately after IS-IS resolves the next-hop for the loopback in this case. Sadly, it doesn't look like my ancient lab switches support IS-IS for IPv6 - so I'll add OSPFv3 for the next topic- actually using Clos in a datacenter network

bgp-rr0-s0#show ip bgp sumBGP router identifier 10.6.0.240, local AS number 65000BGP table version is 3, main routing table version 32 network entries using 234 bytes of memory4 path entries using 208 bytes of memory3/1 BGP path/bestpath attribute entries using 420 bytes of memory0 BGP route-map cache entries using 0 bytes of memory0 BGP filter-list cache entries using 0 bytes of memoryBGP using 862 total bytes of memoryBGP activity 4/0 prefixes, 6/0 paths, scan interval 60 secsNeighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd10.6.0.0 4 65000 8 7 3 0 0 00:03:03 110.6.0.1 4 65000 6 5 3 0 0 00:01:51 1FD00:6:: 4 65000 0 0 0 0 0 never ActiveFD00:6::1 4 65000 0 0 0 0 0 never Active

Configurations generated by this lab, if you want to replicate are here.