rhira777

Posted on Aug 15, 2022

[SONiC Virtual Switch] EVPN-VXLANv4 / BGP unnumbered

#sonic #bgp #vxlan #evpn

Summary

I tried BGP unnumbered and the combination with EVPN-VXLAN in the SONiC on CML environment that we had previously built.

In this case, the virtual switch version of SONiC.202205 was configured with a minimum of communication checks in mind, and were able to achieve IPv4 forwarding using BGP unnumbered and L2 forwarding using EVPN-VXLAN.

The following is a record of that.

The main information that I was referring to is as follows:

[Edgecore SONiC] BGP Unnumbered
https://support.edge-core.com/hc/en-us/articles/900002377366--Edgecore-SONiC-BGP-Unnumbered

[Edgecore SONiC] EVPN L2 VxLAN
https://support.edge-core.com/hc/en-us/articles/900002720523--Edgecore-SONiC-EVPN-L2-VxLAN

[Edgecore SONiC] VLAN & Inter-VLAN Routing
https://support.edge-core.com/hc/en-us/articles/900000210426-VLAN-Inter-VLAN-Routing

SONiC USER MANUAL
https://github.com/sonic-net/SONiC/blob/master/doc/SONiC-User-Manual.md

SONiC Command Line Interface Guide
https://github.com/sonic-net/sonic-utilities/blob/master/doc/Command-Reference.md

Vxlan SONiC
https://github.com/sonic-net/SONiC/blob/master/doc/vxlan/Vxlan_hld.md

Detail of what I did

Network diagram(Topology)

Network information

node	hostname	Loopback0	ASN
sonicvsvm-0	R001	10.0.0.1/32	65001
sonicvsvm-1	R002	10.0.0.2/32	65001
sonicvsvm-2	R011	10.0.0.11/32	65000
sonicvsvm-3	R012	10.0.0.12/32	65000
sonicvsvm-4	R013	10.0.0.13/32	65000
sonicvsvm-5	R014	10.0.0.14/32	65000
sonicvsvm-6	R101	10.0.0.101/32	65001
sonicvsvm-7	R102	10.0.0.102/32	65001

node	hostname	VLAN	VNI
sonicvsvm-6	R101	2	5002
sonicvsvm-7	R102	2	5002

node	interface	MAC address	VLAN	IP address	VRF
csr1000v-0	Gi2	0000.0000.0101	untag	192.168.2.101	101-2
	Gi3	0000.0000.0102	untag	192.168.2.102	102-2

SONiC OS configuration (Excerpts)

save and reload (execute as appropriate)

node	command
sonicvsvm-0~7	sudo config save -y
	sudo config reload -fy

transit link: IPv6 link-local

node	command
sonicvsvm-0~5	sudo config ipv6 enable link-local
sonicvsvm-6/7	sudo config interface ipv6 enable use-link-local-only Ethernet0
	sudo config interface ipv6 enable use-link-local-only Ethernet4

loopback: IPv4

node	command
sonicvsvm-0	sudo config interface ip add Loopback0 10.0.0.1/32
sonicvsvm-1	sudo config interface ip add Loopback0 10.0.0.2/32
sonicvsvm-2	sudo config interface ip add Loopback0 10.0.0.11/32
sonicvsvm-3	sudo config interface ip add Loopback0 10.0.0.12/32
sonicvsvm-4	sudo config interface ip add Loopback0 10.0.0.13/32
sonicvsvm-5	sudo config interface ip add Loopback0 10.0.0.14/32
sonicvsvm-6	sudo config interface ip add Loopback0 10.0.0.101/32
	sudo config interface ip add Loopback0 10.255.0.101/32 secondary
sonicvsvm-7	sudo config interface ip add Loopback0 10.0.0.102/32
	sudo config interface ip add Loopback0 10.255.0.102/32 secondary

VLAN

node	command
sonicvsvm-6/7	sudo config vlan add 2
	sudo config vlan member add -u 2 Ethernet8

VXLAN (EVPN compatible)

node	command
sonicvsvm-6	sudo config vxlan add VXLAN 10.255.0.101
sonicvsvm-7	sudo config vxlan add VXLAN 10.255.0.102
sonicvsvm-6/7	sudo config vxlan evpn_nvo add NVO VXLAN
	sudo config vxlan map add VXLAN 2 5002

FRRouting configuration (Excerpts)

sonicvsvm-0/1

no router bgp 65100
router bgp 65001
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 neighbor SERVICE peer-group
 neighbor SERVICE remote-as internal
 neighbor SERVICE update-source Loopback0
 neighbor TRANSPORT peer-group
 neighbor TRANSPORT remote-as external
 neighbor TRANSPORT capability extended-nexthop
 neighbor Ethernet0 interface peer-group TRANSPORT
 neighbor Ethernet4 interface peer-group TRANSPORT
 neighbor Ethernet8 interface peer-group TRANSPORT
 neighbor Ethernet12 interface peer-group TRANSPORT
 bgp listen range 10.0.0.0/8 peer-group SERVICE
 address-family ipv4 unicast
  redistribute connected
  neighbor TRANSPORT activate
  neighbor TRANSPORT allowas-in
 exit-address-family
 address-family l2vpn evpn
  neighbor SERVICE activate
  neighbor SERVICE route-reflector-client
 exit-address-family
exit

sonicvsvm-2/3/4/5

no router bgp 65100
router bgp 65000
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 neighbor TRANSPORT peer-group
 neighbor TRANSPORT remote-as external
 neighbor TRANSPORT capability extended-nexthop
 neighbor Ethernet0 interface peer-group TRANSPORT
 neighbor Ethernet4 interface peer-group TRANSPORT
 neighbor Ethernet8 interface peer-group TRANSPORT
 address-family ipv4 unicast
  redistribute connected
  neighbor TRANSPORT activate
  neighbor TRANSPORT allowas-in
 exit-address-family
exit

sonicvsvm-6/7

no router bgp 65100
router bgp 65001
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 neighbor SERVICE peer-group
 neighbor SERVICE remote-as internal
 neighbor SERVICE update-source Loopback0
 neighbor TRANSPORT peer-group
 neighbor TRANSPORT remote-as external
 neighbor TRANSPORT capability extended-nexthop
 neighbor 10.0.0.1 peer-group SERVICE
 neighbor 10.0.0.2 peer-group SERVICE
 neighbor Ethernet0 interface peer-group TRANSPORT
 neighbor Ethernet4 interface peer-group TRANSPORT
 address-family ipv4 unicast
  redistribute connected
  neighbor TRANSPORT activate
  neighbor TRANSPORT allowas-in
 exit-address-family
 address-family l2vpn evpn
  neighbor SERVICE activate
  advertise-all-vni
 exit-address-family
exit

Confirmation of operation

csr1000v-0

HOST#ping vrf 101-2 192.168.2.102 source 192.168.2.101 repeat 3
Type escape sequence to abort.
Sending 3, 100-byte ICMP Echos to 192.168.2.102, timeout is 2 seconds:
Packet sent with a source address of 192.168.2.101
.!!
Success rate is 66 percent (2/3), round-trip min/avg/max = 5/5/6 ms
HOST#
HOST#!#ping:ok

csr1000v-0 ~ sonicvsvm-7
sonicvsvm-5 ~ sonicvsvm-7

Supplemental information and others

Status of IPv4 FIB

In the preparation process, IPv4 routes exchanged by BGP were being imported, but IPv4 communication for those routes was failing. In this case, the entries displayed in "show ip route" were not displayed in "ip route show".
This problem was resolved by reloading once.

before reloading

admin@R101:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>r 10.0.0.1/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.2/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.11/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:05:53
B>r 10.0.0.12/32 [20/0] via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:05:53
B>r 10.0.0.13/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
B>r 10.0.0.14/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19
C>* 10.0.0.101/32 is directly connected, Loopback0, 10:02:07
B>r 10.0.0.102/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:04:19
  r                      via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:04:19

admin@R101:~$
admin@R101:~$ ip route show
240.127.1.0/24 dev docker0 proto kernel scope link src 240.127.1.1 linkdown
admin@R101:~$
admin@R101:~$ ping 10.0.0.102 -I 10.0.0.101 -c 3
PING 10.0.0.102 (10.0.0.102) from 10.0.0.101 : 56(84) bytes of data.

--- 10.0.0.102 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2027ms

admin@R101:~$

after reloading

admin@R101:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B>* 10.0.0.1/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
  *                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.2/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
  *                    via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.11/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:30
B>* 10.0.0.12/32 [20/0] via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:30
B>* 10.0.0.13/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:25
  *                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:25
B>* 10.0.0.14/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:17
  *                     via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:17
C>* 10.0.0.101/32 is directly connected, Loopback0, 00:06:34
B>* 10.0.0.102/32 [20/0] via fe80::5054:ff:fe1c:514a, Ethernet0, weight 1, 00:01:25
  *                      via fe80::5054:ff:fe1d:5a36, Ethernet4, weight 1, 00:01:25

admin@R101:~$
admin@R101:~$ ip route show
10.0.0.1 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.2 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.11 nhid 227 via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 proto bgp src 10.0.0.101 metric 20
10.0.0.12 nhid 228 via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 proto bgp src 10.0.0.101 metric 20
10.0.0.13 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.14 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
10.0.0.102 nhid 226 proto bgp src 10.0.0.101 metric 20
        nexthop via inet6 fe80::5054:ff:fe1c:514a dev Ethernet0 weight 1
        nexthop via inet6 fe80::5054:ff:fe1d:5a36 dev Ethernet4 weight 1
240.127.1.0/24 dev docker0 proto kernel scope link src 240.127.1.1 linkdown
admin@R101:~$
admin@R101:~$ ping 10.0.0.102 -I 10.0.0.101 -c 3
PING 10.0.0.102 (10.0.0.102) from 10.0.0.101 : 56(84) bytes of data.
64 bytes from 10.0.0.102: icmp_seq=1 ttl=61 time=3.81 ms
64 bytes from 10.0.0.102: icmp_seq=2 ttl=61 time=3.29 ms
64 bytes from 10.0.0.102: icmp_seq=3 ttl=61 time=3.05 ms

--- 10.0.0.102 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 3.045/3.381/3.806/0.316 ms
admin@R101:~$

Status of bridge table

After successful L2 communication, the EVPN Route Type 2 routes exchanged by BGP were imported, but their entries were not displayed in output such as "show mac".
In this environment, it is possible that this is a display problem or that unicast communication also success as BUM forwarding. It may be possible to proceed to isolate this point, for example, by adding NVEs.

sonicvsvm-6

admin@R101:~$ #ping:ok
admin@R101:~$
admin@R101:~$ show mac
  No.    Vlan  MacAddress         Port       Type
-----  ------  -----------------  ---------  -------
    1       2  00:00:00:00:01:01  Ethernet8  Dynamic
Total number of entries 1
admin@R101:~$
admin@R101:~$ vtysh

Hello, this is FRRouting (version 8.2.2).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

R101#
R101# show bgp l2vpn evpn
BGP table version is 9, local router ID is 10.0.0.101
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]
EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 10.0.0.101:1
*> [2]:[0]:[48]:[00:00:00:00:01:01]
                    10.255.0.101                       32768 i
                    ET:8 RT:65001:5002
*> [3]:[0]:[32]:[10.255.0.101]
                    10.255.0.101                       32768 i
                    ET:8 RT:65001:5002
Route Distinguisher: 10.0.0.102:1
* i[2]:[0]:[48]:[00:00:00:00:01:02]
                    10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8
*>i                 10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8
* i[3]:[0]:[32]:[10.255.0.102]
                    10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8
*>i                 10.255.0.102             0    100      0 i
                    RT:65001:5002 ET:8

Displayed 4 out of 6 total prefixes
R101#

Top comments (1)

Arthur • Sep 1 '23 • Edited

Hi, rhira777. This is really amazing and helps me a lot! And I have a question that how does R101 forward data packet to 192.168.2.101 when ping from 192.168.2.102? I mean through what way R101 knows mac ip and port binding. Thanks for the help!

DEV Community