DEV Community: Amr ElHusseiny

Linux Networking Part 1 : Kernel Net Stack

Amr ElHusseiny — Wed, 04 Jan 2023 07:58:40 +0000

Original article on my blog : https://amrelhusseiny.github.io/blog/004_linux_0001_understanding_linux_networking/004_linux_0001_understanding_linux_networking_part_1/

Introduction to series

1st thing 1st, its very handy to download the uncompiled Linux Kernel code from here https://www.kernel.org.

In this series, we will be exploring the way networking in the server world and how it evolved from using the traditional Linux Kernel Networking stack to network virtualization using OVS and towards handling the load of Telco using NFV and SR-IOV.

This article in a pinch

The following diagram only shows in a brief what happens to a packet in the Linux Kernel if you would like to glance by, but for more in depth and handy details, keep on reading.

Part 1 : Linux Network Stack

In this article, we are the basic flow IPv4/TCP traffic in Linux Kernel following Jiri Binc's talk in DevConf CZ 2018, where he beautifully laied out the packet flow for the whole 7 layers of OSI in the Linux Kernel.
Before we get to the flow path, there is some helper tools and concepts we should be familiar with:

1) Ring Buffers

At bootup of NIC devie and loading its driver module by the Kernel, the drivers starts by allocating Rx(Recieve) and Tx(Transmission) queues or buffers refered to as Ring Buffers in the device memory, usually the DMA part of the kernel space of memory, you can check the Max and configured sizes of these buffers :

# ethtool -g INTERFACE_NAME 
Ring parameters for ens192:
Pre-set maximums:
RX:             4096    <<<<<<<<<<<<<<<, Max size in bytes
RX Mini:        2048
RX Jumbo:       4096
TX:             4096    <<<<<<<<<<<<<<<, Max size in bytes
Current hardware settings:
RX:             1024    <<<<<<<<<<<<<<<, Configured size in bytes
RX Mini:        128
RX Jumbo:       512
TX:             512     <<<<<<<<<<<<<<<, Configured size in bytes

In earlier versions of the kernel, a packet arriving to those buffers, would trigger a hardware interrupt to the CPU per packet, which is very intrusive, but thankfully NAPI was introduced to help with this issue, below you will get to know it more.

2) Socket Buffers (sk_buff)

sk_buff (Socket buffer), linux interacts with a packet/cell/segment or whaterver the recieved network unit using the Socket Buffer (sk_buff), and each sk_buff's data and metadata (Headers) are treated separatly so kernel does not have to move the packet in memory, socket buffer acts like bucket, they hold the segment data until its done being processed, socket buffers donot get destroyed with packets, they are released and reallocated to new packets, also when they are utilized, the CPU creates new sk_buffs (new buckets) to handle the additional traffic.

Side Note: "sk_buff" and "skb" are interchangable, while you will find skb widely used in the kernel code
sk_buff consists of (shown in figure below):

packet buffer (sk_buff data) : a place where the actual packet and data are stored in kernel space memory (DMA memory space - will explore the concept of DMA below), pointed to using the sk_buff struct, the size of the allocated SKB is equal to TCP MSS+Headroom to allow for MSS to change according to connection and user modifications.
sk_buff struct (Socket Buffer Metadata) : MetaData about the packet stored in packet buffers (Data), which includes pointers and values, they look as follows, keep in mind these are just a part of the sk_buff struct, to read the details of sk_buff.h struct, you can find it here:

1) Interface (input_dev) : refers to the interface name the packet arrived at.
2) Protocol : IPv4, IPv6 and so on.
3) Head : pointer to the start of the sk_buff, which actually starts with an empty space giving headroom for extra headers, like a VLAN tag for instacnce.
4) Data : the data pointer does not indicate the start of data, rather its used dynamically in the stack functions to pop and push headers, so fo example in the kernel when you say pop Ethernet header, all that actually happens is that the data pointer moves to the start of the IP Header, so no headers are physically popped in the memory.
5) Tail : indicates points to the end of the data part and start of the empty part of the sk_buffer, again this empty part is used for diffirent sized packets, since the sk_buffer is sized according to the MTU configured.
6) End : points to the end of the sk_buff in memory .
7) MAC & IP & TCP header pointers are always stored in the sk_buff metadata, enabling to call them directly without needing to do poping and pushing actions on the sk_buff
8) cloned : the head of the SKB may be cloned, not the data though.

Note : the packet does not get duplicated in Kernel, actual packet data stays in the packet buffers, with each clone or copy, the packet buffer stays intact, instead a new sk_buff (AKA SKB) is created, so new Metadata pointing to the existing packet buffer, although no copying of the packet in the Kernel, the packet is copied it reaches the application and the SKB is released.

3) Kernel Interrupts (IRQ vs SoftIRQ)

Simply, interrupts are used to stop the CPU from what it is doing and work on the interrupter's part instead, there are models of interrupts, each include many many types, but following you can see there are two categories for these interrupts.

Top-Half Interrupts (Hardware Interrupt) : These kind of interrupts is very costly, and as a result the Interrupt handler masks it after 1st use, and then after that the NIC driver starts to use the SoftIRQ (Software interrupt) instead which can be interrupted by itself, you can observe these interrupts :

$ cat /proc/interrupts
## These are the hardware interrupts, including the IRQ ID , the CPU and the number of interrupts of this type that was triggered .
           CPU0       CPU1       CPU2       CPU3       CPU4           
  0:         28          0          0          0          0  IO-APIC   2-edge      timer
  1:          0          0          0          0          0  IO-APIC   1-edge      i8042
  8:          0          0          0          0          0  IO-APIC   8-edge      rtc0
  9:          0          0          0          0          0  IO-APIC   9-fasteoi   acpi
 12:          0          0          0          0          0  IO-APIC  12-edge      i8042
 14:          0          0          0          0          0  IO-APIC  14-edge      ata_piix
 15:          0          0          0          0          0  IO-APIC  15-edge      ata_piix

Each interrupt (Hardware Interrupt) is identified by a vector which is a one byte identifier, ranging 0-255, from 0-31 are what are called Exceptions (Non-Maskable) interrupts, range 32-47 are maskable interrupts, from 48 to 255 are allocated to Software interrupts (SoftIRQ).

Quickly, there are 3 types of Hardware interrupts you will face in the output above, MSI-X, MSI, and legacy IRQs, in a brief MSI stands for Message Signaled Interrupts which replaces the old way of handling interrupt using single pysical pin in the CPU socket for each device.
You can read about MSI and other types of hardware interrupts here https://en.wikipedia.org/wiki/Message_Signaled_Interrupts.
Also for more info about Hardware interrupts, check this great paper "Linux Interrupts : The basic concepts".

Bottom-Half Interrupts (Software Interrupt) : SoftIRQs runs a queue per CPU, you can find them in the ps output, formated as [ksoftiqd/CPU_Number], these queues polls the device driver for processing traffic, instead of device (NIC) hardware interrupting the CPU each time it recieves traffic, you can see recieve and transmission queues :

# SoftIRQs queues process 
$ ps aux | grep ksoftirqd
root          14  0.0  0.0      0     0 ?        S    22:56   0:00 [ksoftirqd/0]
root          23  0.0  0.0      0     0 ?        S    22:56   0:00 [ksoftirqd/1]
root          29  0.0  0.0      0     0 ?        S    22:56   0:00 [ksoftirqd/2]
root          35  0.0  0.0      0     0 ?        S    22:56   0:00 [ksoftirqd/3]
root          41  0.0  0.0      0     0 ?        S    22:56   0:00 [ksoftirqd/4]

# Monitoring the Rx and Tx buffers :
$ watch -n1 grep RX /proc/softirqs
Every 1.0s: grep RX /proc/softirqs
      NET_RX:          0          2          0        122  

$ watch -n1 grep TX /proc/softirqs
Every 1.0s: grep TX /proc/softirqs
NET_TX:          0          0          0          0

3) Other quick concepts

DMA (Direct Memory Access) : NIC devices are PCIe devices, previously in order to write something to the memory, they had to interrupt the CPU, so the CPU copies the packet to a register and then write it to memory, DMA provides non CPU resources with direct access to the memory without interrupting the CPU, example, when a NIC (Network Interface Card - Hardware) has an Ethernet segment that it wants to write to memory, it uses DMA to directly write it to the Memory, without wasting valuable CPU cycles, these DMA writing calls coming from the NIC are redirected by the NorthBridge on the moterhboard to the RAM instead of the CPU intresting to read more about memory allocation in this article "Allocating Memory for DMA in Linux - by BLAKE RAIN".
Ring Buffers : the NIC and the NIC drivers share a TX and RX ring buffers which basically consists of pointers to the location of packet buffers in the memory, they donot contain data, they are only memory pointers.
Top half and Bottom half : when a packet is DMAed by the NIC to the memory (DMA is a place in Kernel Memory space where NIC card has access to without need for CPU), an Intrupet (SoftIRQ - Soft Interrupt Request) is sent by the NIC to the CPU informing it that a new packet arrived and waiting to be processed, Top half refers to the actions taken at 1st by the CPU, so instead of stopping everything to process this packet, which can be intrusive, it just acknowledges the intrupt and schedules the Bottom half (whcih is the rest of action that will be taken to process the packet) for later.
Context Switching : the process of moving between the UserSpace Context and the KernelSpace context for a process, which consumes CPU cycles.
System Calls : Simply put, system calls are used by user in UserSpace to request service from KernelSpace,
NAPI - for recieved traffic : (New API - need Hardware support), Extension to device drivers designed for network devices to lower the number of interrupts for recieving packets, which comes into effect when there is a huge amount of packets recieved, but it still works in conjunction with normal intrupption process, it also helps with throttling traffic, if the NIC is recieving too much traffic, the NAPI performs dropping the packets on the NIC level without the need to alert/interrupt the kernel, NAPI is only effective on packet recieve events .
- SoftIRQ : The “softIRQ” system in the Linux kernel is a system that kernel uses to process work outside of the device driver IRQ context, device drivers IRQ (Interrupts) are normally of the highest priority for the Linux kernel, and they pause any other types of intrrupts when they arrive, KsoftIRQ is the queue initiated as a thread per CPU very early on in the Kernel, they handle the SoftIRQ queueing, you can see these queues counters using $ cat /proc/softirqs
- ISR (Interrupt Service Routine) : A function in the Kernel responsible for figuring out the nature of the interrupt and what actions to be executed after which the CPU resumes to process previously paused processes.

Network flow in brief

Figure B : simple explanation of what is being allocated and how packet goes though Kernel :
1) Very early at Kernel boot up, the CPU allocates packet buffers (RX and TX buffers), and build file descriptors.
2) CPU informs the NIC that new descriptors has been created for the NIC to start using.
3) DMA (Direct Access Memory) fetches descriptors.
4) Packet arrives at NIC.
5) DMA Writes the packet to the RX Ring buffer.
6) NIC informs the driver which informs the CPU that new traffic is ready to be processed using Hardware Interrupt (IRQ).
8) After the 1st Hardware interrupt, the Interrupt handler masks it, and instead the driver utilizes the use of Software Interrupt (SoftIRQ) which is much less costly to the CPU (Hardware Interrupts cannot be interrupted which is very costly for the CPU).
9) The SoftIRQ invokes the NAPI subsystem (Wakes up), which calls the NIC Driver's Polling function.
9) CPU process the incoming packets.
10) After a certain time of the SoftIRQs budget runs out, the NAPI system gets back to sleep, if the budget of the SoftIRQs runs out, the CPU moves on to the next task, and the time_squeezed counter in the /proc/net/softnet_stats is incremented by 1.

At NIC initiation, the driver does teh following :
1) Allocates Rx & Tx queues in memory (DMA space).
2) Enable NAPI, which is off by default.
3) Register an Interrupt Handler.
4) Enable Hardware interrupts

I would recommend for a detailed description of the flow, to check this link

Keywords

File Descriptor : is a number the operating system uses to identify an open file

- TSS (Tuple Space Search) : this is used by OVS for the Second Level table which is dpcls basicaly it depends on hash matching to existing tuple, meaning when you are able to match on a tuple of same values in each packet , you create a hash to match it and forward based on that hash, instead of looking up each value separatly .

Commands Summary

Description	command
Show Interrupt counters (IRQs and SoftIRQs)	`$ cat /proc/interrupts`
Show SoftIRQ Processes	`$ ps aux \
Monitor the RX Buffer per CPU (Rx & TX)	{% raw %}`$ watch -n1 grep RX /proc/softirqs`
	`$ watch -n1 grep TX /proc/softirqs`
To see the number of packets dropped by the NIC	`$ cat /sys/class/net/ens192/statistics/rx_missed_errors`

References

gNMI Network Automation (Part 3) : gNMI Telemetry, Telegraf & InfluxDB

Amr ElHusseiny — Sat, 19 Nov 2022 23:20:01 +0000

gNMI Network Automation (Part 3) : gNMI Telemetry, Telegraf & InfluxDB

Introduction

This article is based on Damien Garros's talk in NANOG 77 "Getting started with modern time series database".

We are used to monitoring network devices using SNMP by pulling devices statistics through an RRD server (example Cacti), but a new approach (not so new) is replacing SNMP by relying on Streaming (Push) telemetry from device to server instead of pulling, by using gNMI (Network gRPC protocol originally developed by Google, now widely adopted by vendors).

Summary of steps will be taken :
1) Setup Containerlab and prepare the IOS-XR image using Part1.
2) Get to know a little bit abouyt gNMI and gRPC and how to read configuration templates for OpenConfig and Vendor Specific in Part2 .
3) Start our testing Network topology.
4) Configuring 2 IOS-XR PE/P routers using gNMI.
5) Pulling and configuring both Telegraf and InfluxDB containers.

Note : All Lab files used in this article, you can find in my Github here .

Brief about topology

1st, we have 2 containerlab topology files, one for the Network topology that we are will be collecting telemtry data from, and one for the Telegraf (Collector) and InfluxDB (Time Streamed Data Database) .

The network topolgy consists of :

2 IOS-XR 6.5.1 PE/P routers : running a full L3VPN Backbone protocols, IS-IS/MPLS/LDP/MP-BGP and will be configured using gNMI configuration files you can find ready in the github page, I chose to confgiure them using gNMI so you can check how to use gNMI also for configuration management not just for Telemetry.
2 Open FRRouting containers runing as CE routers, configured with eBGP, configuration files are loaded onto them while initiating using Containerlab, so nothing to be done there, although its cool to get to know FRR, if you want to log onto the FRR container, you can use '# docker exec -it CE_NAME vtysh', you will find its prompt is like Cisco's IOS.
2 Linux containers connected to CEs, will be used to generate traffic using IPerf3.

Running the topology

1) Initiate the topology using Containerlab :
$ containerlab deploy -t 04_topology.clab.yml
The containlab template loks like follows, you can also find it here.



name: 04_topology

topology:
  nodes:
    # CEs using  RRouting open image
    CE1:
        kind: linux
        image: frrouting/frr:v7.5.1
        binds:  # Binds used to copy files to the created containers at start
            # 1) FRR uses the Dameon file to check whiches protocols to spin up as it runs as seprate process
            # same daemons file will be used on both CEs as they are running the same protocols.
            - template_config/04_CE1_FRR_daemons:/etc/frr/daemons
            # 2) Router configuration is loaded configuring interfaces, and eBGP session.
            - template_config/04_CE1_FRR_conf:/etc/frr/frr.conf
    CE2:
        kind: linux
        image: frrouting/frr:v7.5.1
        binds: 
            - template_config/04_CE1_FRR_daemons:/etc/frr/daemons
            - template_config/04_CE2_FRR_conf:/etc/frr/frr.conf
    # Backbone PEs (ISIS/MP-BGP/MPLS)
    PE1:
        kind: vr-xrv9k
        # To be able to use this image, you need to check Part 1 and 2 so you can create an IOS-XR container.
        image: vrnetlab/vr-xrv9k:7.2.1
    PE2:
        kind: vr-xrv9k
        image: vrnetlab/vr-xrv9k:7.2.1
    PC1:
        kind: linux
        image: praqma/network-multitool:latest
        # Installing IPerf3 at startup
    PC2:
        kind: linux
        image: praqma/network-multitool:latest

  links:
    - endpoints: ["PC1:eth1", "CE1:eth2"]
    - endpoints: ["CE1:eth3", "PE1:eth3"]
    - endpoints: ["PE1:eth2", "PE2:eth2"]
    - endpoints: ["PE2:eth3", "CE2:eth3"]
    - endpoints: ["CE2:eth2", "PC2:eth1"]

The IOS-XR images will take 10 to 15 minutes, you can follow its progressusing Docker logs, you should see the following message when its fully started up :



$ docker logs -f clab-04_topology-PE1
.......
2022-11-19 20:01:16,258: launch     INFO     Startup complete in: 0:08:11.193191

2) FRR CEs are already configured (configuraiton under the template_config/04_CE1_FRR_conf & template_config/04_CE2_FRR_conf), now to configure the Linux containers interfaces and default gateway, create or run the file on Github, simply configuring the Linux interfaces :



#!/bin/sh
sudo docker exec clab-04_topology-PC1 ip link set eth1 up
sudo docker exec clab-04_topology-PC1 ip addr add 10.0.10.2/24 dev eth1
sudo docker exec clab-04_topology-PC1 ip route add 10.0.20.0/24 via 10.0.10.1 dev eth1
sudo docker exec clab-04_topology-PC1 apk add iperf3 

sudo docker exec clab-04_topology-PC2 ip link set eth1 up
sudo docker exec clab-04_topology-PC2 ip addr add 10.0.20.2/24 dev eth1
sudo docker exec clab-04_topology-PC2 ip route add 10.0.10.0/24 via 10.0.20.1 dev eth1
sudo docker exec clab-04_topology-PC2 apk add iperf3

3) To configure the PEs, using gNMI, make sure to have gNMIC tool installed , then run the following 2 commands under the gnmi_config directory :
Please find the configuration files in the following links:

04_xrv_pe1_gnmi.yml
04_xrv_pe2_gnmi.yml ```bash

$ gnmic -a clab-04_topology-PE1 --insecure -u admin -p admin -e json_ietf set --request-file 04_xrv_pe1_gnmi.yml
$ gnmic -a clab-04_topology-PE2 --insecure -u admin -p admin -e json_ietf set --request-file 04_xrv_pe2_gnmi.yml

For this gNMI configuration file, you can check the previous article where you will find a detailed description of each configuration part.

Now, the testing network topology is readymoving to our main scope, Telemetry.

## Our telemetry setup now
Before we get into the details, lets start by pulling the needed containers : 
```bash


docker pull telegraf
docker pull influxdb

We will use containerlab replacing Docker Compose to spin up the containers wth their configuration, before running the container, we will have a brief about the tools.

Telegraf (Collector)

There is 2 ways in subscribing to gNMI telemtry:
1) Dial-Out : you configure your network device to initiate the TCP session to the collectore and stream out the configured telemetry points, not recommended as it means you need to configure the subscription Metrics on each node, and in case you need to do a change, you need to roll out these changes to all devices.
2) Dial-In : We use this method in our Lab , we only allow gNMI service/port on our IOS-XR devices, then we do all the sunscription configuration on our gNMI Collector (Telegraf).

Telegraf is the collector agent which will Dial-In/subscribe to a Metric on the device, Telegraf has a plugin for gNMI developed by Cisco initially that enables configuration of gNMI telemtry, since Telegraf is not only usef for gNMI .

To understand configuraiton of the Telegraf Container, you have :
1) Agent : In this configuration, you would specify interval in which measurements are taken, output batch size, since telegraf aggregates the metrics before sending them out to the Influx database, so you configure the batch size, and the buffer limit for it, so when it momentarliy lose connection to the database, it can cash those metrics until connection is restored and then send them all at once, and so on, here am using the default generated configuration.



[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = "0s"
  hostname = "telegraf"
  omit_hostname = false

3) Input plugins : are responsible for transforming incoming data stream from device format (Example : Json , Binary, Protobuffs, etc) to Telegraf Metics format ( Measurement Name, Tages, Fields, Timestamp ), in our case Cisco developed a gNMI Input plugin which has been merged to the main Git for Telegraf which subscribes to gNMI Paths.
For the path that I used , please refer to my previous gNMI guide, you can use either Cisco's Yang Paths or OpenConfig's , here I will be using OpenConfig's, this will make it easier to use across many vendors.



[[inputs.gnmi]]
    addresses = ["clab-testing_topology-PE1:57400"]
    username = "admin"
    password = "admin"
    redial = "10s"

Now we can include multiple subscriptions under the same gNMI input module, you can name each one with your preferred description, for Paths, you can use either the OpneConfig ones or the Vendor specific ones, for me i prefer to use gNMIC tool to check what is the output for each module before deciding to use which path, for details on how to use gNMIC, you can refer to my previous article.
We are creating 2 subscriptions in this example.
1st example is for interface counters, which we want to take on a "Sample" basis in time intervals:



[[inputs.gnmi.subscription]]
    name = "interfaces_counters"
    origin = "openconfig-interfaces"
    #path = '/interfaces/interface[name="GigabitEthernet0/0/0/1"]/state/counters'
    path = '/interfaces/interface/state/counters'
    subscription_mode = "sample" # you can find the modes in the gNMI standard
    # Setting sample to 30 Seconds, because on Cisco IOS-XR interval is minimum at 30 Seconds
    sample_interval = "30s"
    heartbeat_interval = "60s"

Before our next article and Grafana integration, InfluxDB provides a very good WebUI just like Grafana's to see your data collected, just to show you after spinning up our configuration, Interfsces output with some Iperf traffic generation from our linux edge containers looks as follows in Influx dashboard :

2nd example, we want to get the IS-IS Adjacency table but in structured way instead of relying on the old SSH/CLI parsing way, you can export/Putput the data in Json format to your script for example, this subscription wont create a graph, instead, its more for Data gathering, this shows you the extensibilty of Telemetry in gNMI, not just for counters, but it can be used for Operational data :



[[inputs.gnmi.subscription]]
    name = "isis_advacency"
    origin = "Cisco-IOS-XR-clns-isis-oper"
    path = '/isis/instances/instance[instance-name="default"]/neighbors'
    subscription_mode = "on_change" # you can find the modes in the gNMI standard
    heartbeat_interval = "60s"

3) Processor & Aggregator plugins: processors are used to filter, decorate, or trasform input as its coming in, unlike aggregators, which are applied over a period of inputs to produce outputs like mean, minimum, maximum and count.
4) Output Plugins: Just like Input plugins, this shapes data towards the external resource that will store or analyze the data, in our case, we will be using InfluxDB, which provides and Output Plugin for Telegraf.



[[outputs.influxdb_v2]]
   urls = ["http://clab-03_ansible_iosxr_linux_telemetry-influxDB:8086"]
   token = "Df-veQgYbF i m40KZfT×q1zHUN3- - IRV4WTHdfXsK_ SUNCiX6171pjGZMHOgYl jmgG-BiwMprBLocpX4L82xg=="
   organization = "test_org"
   bucket = "test_bucket"

You can find the fullconfiguration file fo Telegraf here, we will use it later while spinning up the container.

InfluxDB (Time Streaming Database)

We will use InfluxDB to store the telemetry and collector's data, so we can graph it either using Grafana or in our case, InfluxDB now offers some of Grafana's functionallity, we will use Grafana in the next article.
To configure Influx, its very simple, you just need to set some Env variables as per their official Docker guide , which we will do using ContainerLab.

Running the Telegraf and InfluxDB contianers

Now we will use a second ContainerLab file to initiate the Telegraf+Config and InfluxDB+ENV variables

Note : I kept the Network topology setup separate from Telegraf&InfluxDB t give you the ease of reconfiguring telemtry without having to destroy and redeploy the IOS-XR images which takes a long time to start.

Okey, the 2nd containerlab file looks as follows , link to github :



# 04_telemetry.clab.yaml
name: 04_telemetry

topology:
  nodes:
    # InfluxDB ------------------------------------------------------------------------
    influxDB:
        kind: linux
        image: influxdb:latest
        ports:
            ## This port will be used for Telegraf to connect to 
            - 8086:8086
        env:
            DOCKER_INFLUXDB_INIT_MODE: setup
            DOCKER_INFLUXDB_INIT_USERNAME: influx_super
            DOCKER_INFLUXDB_INIT_PASSWORD: 1020304050607
            DOCKER_INFLUXDB_INIT_ORG: test_org
            DOCKER_INFLUXDB_INIT_BUCKET: test_bucket
            DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: "Df-veQgYbF i m40KZfT×q1zHUN3- - IRV4WTHdfXsK_ SUNCiX6171pjGZMHOgYl jmgG-BiwMprBLocpX4L82xg=="
    # Telegraf ------------------------------------------------------------------------
    telegraf:
        kind: linux
        image: telegraf:latest
        binds:
            - telegraf.conf:/etc/telegraf/telegraf.conf:ro
        ports:
            - 57000:57000

Run
$ containerlab deploy -t 04_telemetry.clab.yaml
You can check no errors on the Telegraf connection to the InfluxDB or to the PE1 gNMI subscription using docker logs :



root@vnet:/home/vnet/github_2# docker logs -f clab-04_telemetry-telegraf
2022-11-19T21:57:36Z I! Using config file: /etc/telegraf/telegraf.conf
2022-11-19T21:57:36Z I! Starting Telegraf 1.24.3
2022-11-19T21:57:36Z I! Available plugins: 222 inputs, 9 aggregators, 26 processors, 20 parsers, 57 outputs
2022-11-19T21:57:36Z I! Loaded inputs: gnmi
2022-11-19T21:57:36Z I! Loaded aggregators: 
2022-11-19T21:57:36Z I! Loaded processors: 
2022-11-19T21:57:36Z I! Loaded outputs: influxdb_v2
2022-11-19T21:57:36Z I! Tags enabled: host=telegraf
2022-11-19T21:57:36Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"telegraf", Flush Interval:10s

After that log into the Linux container clab-04_topology-PC2 , and 1st test connectivity to clab-04_topology-PC1, and then if ping is okey, start IPerf3 server on PC2 :



clab-04_topology-PC2#ping -c 3 10.0.10.2
PING 10.0.10.2 (10.0.10.2) 56(84) bytes of data.
64 bytes from 10.0.10.2: icmp_seq=1 ttl=60 time=19.4 ms
64 bytes from 10.0.10.2: icmp_seq=2 ttl=60 time=1.57 ms
64 bytes from 10.0.10.2: icmp_seq=3 ttl=60 time=1.44 ms
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.438/7.469/19.395/8.433 ms

clab-04_topology-PC2# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

And on clab-04_topology-PC1 lets generate some traffic :



$ iperf3 -c 10.0.20.2 -u -b 1000M
Connecting to host 10.0.20.2, port 5201
[  5] local 10.0.10.2 port 53437 connected to 10.0.20.2 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  98.2 MBytes   824 Mbits/sec  71323  
[  5]   1.00-2.00   sec  95.9 MBytes   804 Mbits/sec  69605  
[  5]   2.00-3.00   sec  98.3 MBytes   824 Mbits/sec  71356  
[  5]   3.00-4.00   sec  98.3 MBytes   825 Mbits/sec  71405  
[  5]   4.00-5.00   sec  98.5 MBytes   827 Mbits/sec  71556  
[  5]   5.00-6.00   sec  98.2 MBytes   823 Mbits/sec  71283  
[  5]   6.00-7.00   sec  97.9 MBytes   821 Mbits/sec  71070  
[  5]   7.00-8.00   sec  98.2 MBytes   824 Mbits/sec  71290  
[  5]   8.00-9.00   sec  98.1 MBytes   823 Mbits/sec  71223  
[  5]   9.00-10.00  sec  98.4 MBytes   825 Mbits/sec  71455  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   980 MBytes   822 Mbits/sec  0.000 ms  0/711566 (0%)  sender
[  5]   0.00-10.26  sec   245 KBytes   196 Kbits/sec  2.803 ms  710324/710498 (1e+02%)  receiver

iperf Done.

You can generate traffic for somewhile and then lets head to InfluxDB to see our telemetry plot.

Graphs on InfluxDB

You are able to login to the Web portal of the InfluxDB from your local network, to get ip of the InfluxDB, you can use Containerlab to inspect all the current running nodes :



containerlab inspect -a | grep influx
| 1 | 04_telemetry.clab.yaml | 04_telemetry | clab-04_telemetry-influxDB | 2797cc685296 | influxdb:latest                 | linux    | running | 172.20.20.8/24 | 2001:172:20:20::8/64 |

Open your web browser to http://172.20.20.8:8086 with user influx_super and password 1020304050607 which we setup in the 04_telemetry.clab.yaml.
Go to Data Explorer , then choose filter as following in snapshot :
1- this shows the interface counters of interface Ge0/0/1, but this accumalates, it does not show the current count only , as on Cisco interface counters just adds up over time, to show the diffirence only , go to step 2.

2- Change view to 'script editor' and search 'filter function' for 'diffirence' function , add it, and remove the line with 'yield mean' now you only can see the increase/decrease of bytes count on the interface.

Lets also have a look at the ISIS adjaceny neibors on PE1 , this is the benefit of gNMI, it provides more than just counters, you can see even the routing table on change.

Thats it, you can tweek the telegraf.conf file to add your telemetry subscriptions which you can get from the yang files described in Part 2 of this guide .

Final thing , to destroy all the created containers, just type $ containerlab destroy -a

I hope you enjoyed this article :) , next will be intergration with Grafana .

References

gNMI Network Automation (Part 2) : gNMI Configuration Cisco's IOS-XR

Amr ElHusseiny — Tue, 08 Nov 2022 18:48:06 +0000

Introduction

In this post we are going to focus on 2 parts:
1- Use gRPC to configure a l3VPN backbone (IS-IS,MPLS,MP-BGP) in a 2 node IOS-XR backbone PE/P routers.
2- 2 customer edge CPEs (Using PreConfigured FRR Nodes) and 2 net-tool Linux containers will be used for testing connectivity (Configured using Bash).
3- Next article , we will use gNMI telemetry for gathering interface data and BGP/ISIS adjacency and collect them using Telegraf/InfluxDB and visualize them using Grafana.

OpenConfig vs Vendor

1st you need to separate gNMI from OpenConfig, gNMI is implemented very maturely and you can find nearly all Network vendors adheres to its standards, this gives you the way to send gRPC calls to the devices, on the other hand OpenConfig implementation widely vary, as you will see in this tutorial which focuses on IOS-XR, some features can be fully configured using OpenConfig schemas/Yangs, and some features mix and match between OpenConfig and Vendor specific Yangs.

For myself, I'd say there is a long way for OpenConfig to provide a uniform way to interact with multiple platforms.

A very good article to check for how OpenConfig implements the concept of VRFs and routing protocol, is to check their official OpenConfig RIB Approach, consider this a prerequsiste before you proceed with the below example, as Openconfig replaces the concept of VRF with a Network Instance, and instead of VRF being one RIB (Table in OpenConfig) with multiple Protocols (OSPF,BGP,..) installing routes into, some vendors do treat all the RIBs/tables under one VRF/Network-instance as one entitiy (RIB/Table), and so all protocol routes are used for best path selection like traditional routing, and some other vendors like Nokia keeps each Protocol table separate under each Network-instance, in this case, you will need to do OpenConfig's Table-Connection between diffirent protocols tables under the same Network-Instance.

How can you figure out whether a vendor is implementing single table RIB or multi-table RIB ?
You can always check the Deviations Yang file under the Yang implementation of the vendor, in our case its not supported on IOS-XR, as the deviations file shows, you can see in output below , that tables and table connections are not supported :



cat cisco-xr-openconfig-network-instance-deviations.yang
  deviation "/oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:table-connections/oc-netinst:table-connection" {
    deviate not-supported;
  }
  deviation "/oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:interfaces/oc-netinst:interface/oc-netinst:config/oc-netinst:associated-address-families" {
    deviate not-supported;
  }
  deviation "/oc-netinst:network-instances/oc-netinst:network-instance/oc-netinst:tables/oc-netinst:table" {
    deviate not-supported;
  }

Here is a more clear representation from the Anatomy of the Network Instance in OpenConfig

Solving the lack of documentation examples

I faced a big issue figuring out how each vendor implemenets the configuration parts of the gNMI tree, and through my exploration i found 2 ways to go about it :

1st Approach : from device CLI

In my IOS-XR example, some parts like the Routing Policies were not implemented in gNMI the same way you would write it in CLI, so what i did is opened a IOS-XR VM configured a route policy as follows :



route-policy ALLOW_ANY
  pass
end-policy

Then in each vendor there is a way to show the configuration in an XML way , and some have a way to show it in the OpenConfig hirarichy like Cisco, you can use an online tool like CodeBeautify to convert XML into YAML :



#show running-config | xml openconfig 
<routing-policy xmlns="http://openconfig.net/yang/routing-policy">
   <policy-definitions>
    <policy-definition>
     <name>ALLOW_ANY</name>
     <config>
      <name>ALLOW_ANY</name>
     </config>
     <statements>
      <statement>
       <name>statement-1667049259886989</name>
       <config>
        <name>statement-1667049259886989</name>
       </config>
       <actions>
        <config>
         <accept-route/>
        </config>
       </actions>
      </statement>
     </statements>
    </policy-definition>
   </policy-definitions>
  </routing-policy>

2nd Approach : Reading Yang files

If you dont have access to a live device, you can read the YANG trees of vendors or OpenConfig which you can clone from Vendor's Yang Models in this link, you can fond your needed path in 2 ways :
1) Tree hirarichy, including variable types and keys using PYang.



\# pyang openconfig-network-instance.yang -f tree
module: openconfig-network-instance
  +--rw network-instances
     +--rw network-instance* [name]
        +--rw name                       -> ../config/name
        +--rw fdb
        |  +--rw config
        |  |  +--rw mac-learning?      boolean
        |  |  +--rw mac-aging-time?    uint16
        |  |  +--rw maximum-entries?   uint16
        |  +--ro state
        |  |  +--ro mac-learning?      boolean
        ...

2) Using gNMIC extract all the paths in the yang Module (Does not include the variable types) but very handy when you want a path to insert into a Get call :



# gnmic path openconfig-network-instance.yang --file openconfig-network-instance.yang --config-only
/bgp/global/afi-safis/afi-safi[afi-safi-name=*]/afi-safi-name
/bgp/global/afi-safis/afi-safi[afi-safi-name=*]/config/afi-safi-name
/bgp/global/afi-safis/afi-safi[afi-safi-name=*]/config/enabled
...

Deviations

Each vendor implementation of gNMI has some deviations (Some parts not compitable with the OpenConfig Standard) which are documented in Yang files, you can find these Yang file in the same directory of the platform's OpenConfig and Vendor specific yangs, for example, listing the Routing Policies yang in the IOS-XR 6.5.1 version.

1) OpenConfig Yangs :



cisco_yang_models/yang/vendor/cisco/xr/651# ll | grep policy | grep openconfig
-rw-r--r--  1 root root  28487 Oct 14 18:28 openconfig-bgp-policy.yang
-rw-r--r--  1 root root   3688 Oct 14 18:28 openconfig-isis-policy.yang
-rw-r--r--  1 root root   4265 Oct 14 18:28 openconfig-policy-types.yang
-rw-r--r--  1 root root  27024 Oct 14 18:28 openconfig-routing-policy.yang

2) Cisco specific implementation for configuration :



cisco_yang_models/yang/vendor/cisco/xr/651# ll | grep policy | grep Cisco | grep cfg
-rw-r--r--  1 root root  12321 Oct 14 18:28 Cisco-IOS-XR-policy-repository-cfg.yang

3) Cisco specific implementation for streaming telemetry :



cisco_yang_models/yang/vendor/cisco/xr/651# ll | grep policy | grep Cisco | grep oper
-rw-r--r--  1 root root   9567 Oct 14 18:28 Cisco-IOS-XR-policy-repository-oper-sub1.yang
-rw-r--r--  1 root root   9532 Oct 14 18:28 Cisco-IOS-XR-policy-repository-oper.yang

4) Deviations - showing what is not supported usig the OpenConfig, in this case, you will need to use the vendor specific Yang :



cisco_yang_models/yang/vendor/cisco/xr/651# ll | grep policy | grep deviations
-rw-r--r--  1 root root   2927 Oct 14 18:28 cisco-xr-openconfig-bgp-policy-deviations.yang
-rw-r--r--  1 root root   1437 Oct 14 18:28 cisco-xr-openconfig-routing-policy-deviations.yang

Reading the Deviation file is important so you are able to figure out which parts of the OpenConfig is not supported by this platform, for examples in "cisco-xr-openconfig-routing-policy-deviations.yang" you will see :
(This is a small sample from the file for readability)



deviation "/rpol:routing-policy/rpol:defined-sets/rpol:neighbor-sets" {

    deviate not-supported;

  }

Example Lab

You can find this Lab's files on my github, we are using ContainerLab to deploy the setup and connect nodes to each other, after we have 3 diffirent node types to configure :
1) Endpoint VMs : uses net-tools containers, configured using basic shell .
2) CEs (FRRrouting) : using Containerlab configuration is deployed with initiation of the nodes, FRR Open routing software does not support gNMI since its normally deployed as a part of a bigger package.
3) PEs (IOS-XRV 6.5.1) : these 2 PE/P will completly configured using gNMI.

Lets focus on our PEs configuration, gNMIC is a great tool/eco-system enable you to use gNMI Yaml configuraiton templates and variable files to simplify the configuration process, in our case , we will deploy configuration as Yaml, but not use Variable files, this will come in a later tutorial.

I Commented each part of the configuration:



# 02_xrv_pe1_gnmi.yml

updates:

  # ================================================================================

  # Interfacece configuration

  # ================================================================================

  # Used the Cisco Yangs for this part as the OpenConfig yangs does not directly configure the interface IP, it has to be configured on 

  # a sub-interface instead

# Interface GigabitEthernet0/0/0/1 Config

  - path: Cisco-IOS-XR-ifmgr-cfg:/interface-configurations/interface-configuration[active=act][interface-name=GigabitEthernet0/0/0/1]

    value:

      description: to_PE2

      Cisco-IOS-XR-ipv4-io-cfg:ipv4-network:

        addresses:

          primary:

            address: 100.100.10.0

            netmask: 255.255.255.254

    encoding: json_ietf

  # Interface GigabitEthernet0/0/0/2 Config

  - path: Cisco-IOS-XR-ifmgr-cfg:/interface-configurations/interface-configuration[active=act][interface-name=GigabitEthernet0/0/0/2]

    value:

      description: to_CE1

      Cisco-IOS-XR-ipv4-io-cfg:ipv4-network:

        addresses:

          primary:

            address: 100.100.20.1

            netmask: 255.255.255.254

    encoding: json_ietf

  # Interface Loopback0 Config

  - path: Cisco-IOS-XR-ifmgr-cfg:/interface-configurations/interface-configuration[active=act][interface-name=Loopback0]

    value:

      description: PE1_Loopback

      # Ciscos weird implementation dictates to enable some of configuration elements, you need to set it with Null,

      # in this case to enable a Loopback, you need to set the interface-virtual to Null, dont worry , the Yang file mentions it

      interface-virtual: [null]

      Cisco-IOS-XR-ipv4-io-cfg:ipv4-network:

        addresses:

          primary:

            address: 100.64.0.1

            netmask: 255.255.255.255

    encoding: json_ietf

# ================================================================================

  # IS-IS 

  # ================================================================================

  # Configuring ISIS using OpenConfig

  - path: openconfig-network-instance:/network-instances/network-instance[name=default]/protocols

    value:

      protocol:

        # Some identifiers and types are supported by the OpenConfig or vendor like openconfig-policy-types:ISIS

        # you will be able to find them in separate Yang Files called types and so on

        # So in this case , the platform may no recognise ISIS, you have to mention the Types Yang it belongs to

        identifier: openconfig-policy-types:ISIS

        name: default # This is the RIB name rather than a VRF name , under each Network-Instance (Global/VRF) you will find default RIB

        isis:

          global:

            config:

              level-capability: LEVEL_2

              net:

              - 49.0001.0064.0000.0001.00

            afi-safi:

              af: 

                - afi-name: openconfig-isis-types:IPV4

                  safi-name: openconfig-isis-types:UNICAST

          interfaces:

            interface:

              - interface-id: Loopback0

                config:

                  passive: true

                afi-safi:

                  af:

                    - afi-name: openconfig-isis-types:IPV4

                      safi-name: openconfig-isis-types:UNICAST

                      config: 

                        enabled: true

              - interface-id: GigabitEthernet0/0/0/1

                afi-safi:

                  af:

                    - afi-name: openconfig-isis-types:IPV4

                      safi-name: openconfig-isis-types:UNICAST

                      config: 

                        enabled: true

# ================================================================================

  # MPLS & LDP

  # ================================================================================

  # Note : since I am using IOS-XR 6.5.1 , only Cisco Yang models supported for MPLS & LDP

  #        Openconfig Yangs are not supported 

  - path: Cisco-IOS-XR-mpls-ldp-cfg:/mpls-ldp

    value:

      enable: [null]

      default-vrf:

        interfaces:

          interface:

            - interface-name: GigabitEthernet0/0/0/1

              enable: [null]

# ================================================================================

  # MP-BGP (We are using point to point peering instead of RR)

  # ================================================================================

  - path: openconfig-network-instance:/network-instances/network-instance[name=default]/protocols

    value:

      protocol:

        - identifier: openconfig-policy-types:BGP

          name: default

          bgp:

            global:

              config:

                as: 65000

                router-id: 100.64.0.1

              afi-safis:

                afi-safi:

                  - afi-safi-name: openconfig-bgp-types:L3VPN_IPV4_UNICAST

                    config:

                      enabled: true

                  - afi-safi-name: openconfig-bgp-types:IPV4_UNICAST

                    config:

                      enabled: true

            neighbors:

              neighbor:

                - neighbor-address: 100.64.0.2

                  config:

                    peer-as: 65000

                  afi-safis:

                    afi-safi:

                      - afi-safi-name: openconfig-bgp-types:L3VPN_IPV4_UNICAST

                        config:

                          enabled: true

                      - afi-safi-name: openconfig-bgp-types:IPV4_UNICAST

                        config:

                          enabled: true

                  transport:

                    config:

                      local-address: 100.64.0.1

# Configuring Next Hop Self will be deployed using Ciscos Yangs

  # As no similar in OpenConfig 

  # Enablig Next-Hop-Self on both IPv4 and VPNv4 Families (No RR used)

  - path: Cisco-IOS-XR-ipv4-bgp-cfg://bgp/instance[instance-name=default]/instance-as[as=0]/four-byte-as[as=65000]/default-vrf/bgp-entity/neighbors/neighbor[neighbor-address=100.64.0.2]/neighbor-afs/neighbor-af

    value:

      - af-name: ipv4-unicast

        activate: 

        next-hop-self: true

      - af-name: vpnv4-unicast

        activate: 

        next-hop-self: true

# ================================================================================

  # VRF Configuration 

  # ================================================================================

  # Following 3 parts can be combined in one call, but i preferred to separate them to be easier to understand 

# Configuring Allow all Route Policy , IOS-XR by default denies all if no Route

  # Policy configured in eBGP session , cannot use the OpenConfig Yang for this policy , got a lot of issues

  - path: Cisco-IOS-XR-policy-repository-cfg:/routing-policy

    value:

      route-policies:

        route-policy: 

          route-policy-name: AcceptAny

          rpl-route-policy: route-policy AcceptAny\n  pass\nend-policy\n

# Create VRF

  - path: openconfig-network-instance:/network-instances

    value:

      network-instance:

        - name: CustomerVRF

          config: 

            name: CustomerVRF

            route-distinguisher: 1:1

# initiate OpenConfigs BGP-IPv4 table under the CustomerVRF network instance

  - path: openconfig-network-instance:/network-instances/network-instance[name=CustomerVRF]

    value:

      config:

        enabled-address-families: openconfig-types:IPV4

      tables:

        table:

          - protocol: BGP

            address-family: IPV4

            config: 

              protocol: BGP

              address-family: openconfig-types:IPV4

# Doing redistribute Connected into VRF , using Ciscos Yangs, not available in OpenConfig: 

  - path: Cisco-IOS-XR-ipv4-bgp-cfg:/bgp/instance[instance-name=default]/instance-as[as=0]/four-byte-as[as=65000]/vrfs/vrf[vrf-name=CustomerVRF]/vrf-global/vrf-global-afs/vrf-global-af[af-name=ipv4-unicast]/connected-routes

    value: null

  # Doing redistribute Static into VRF , using Ciscos Yangs, not available in OpenConfig: 

  - path: Cisco-IOS-XR-ipv4-bgp-cfg:/bgp/instance[instance-name=default]/instance-as[as=0]/four-byte-as[as=65000]/vrfs/vrf[vrf-name=CustomerVRF]/vrf-global/vrf-global-afs/vrf-global-af[af-name=ipv4-unicast]/static-routes

    value: null

# Assinging interfaces to VRF after VRF creation 

  - path: openconfig-network-instance:/network-instances/network-instance[name=CustomerVRF]

    value:

      interfaces:

        interface:

          id: GigabitEthernet0/0/0/2

          config:

            id: GigabitEthernet0/0/0/2

            interface: GigabitEthernet0/0/0/2

# Configuring BGP under VRF : 

  - path: openconfig-network-instance:/network-instances/network-instance[name=CustomerVRF]/

    value:

      protocols:

        protocol:

          - identifier: BGP

            name: default # this points to the primary RIB under the VRF (Table belongs to CustomerVRF Network-Instance)

            bgp:

              global:

                config:

                  as: 65000

                  router-id: 100.100.20.1

                afi-safis:

                  afi-safi:

                    - afi-safi-name: openconfig-bgp-types:IPV4_UNICAST

                      config:

                        enabled: true

              neighbors:

                neighbor:

                  - neighbor-address: 100.100.20.0

                    config:

                      peer-as: 65001

                    afi-safis:

                      afi-safi:

                        - afi-safi-name: openconfig-bgp-types:IPV4_UNICAST

                          config:

                            enabled: true

                    transport:

                      config:

                        local-address: 100.100.20.1

# Applying: (Unable to same using OpenConfig)

  # 1) Import & Export Policy with eBGP Peer (CE)

  # 2) Applying AS-Override (cuz both CEs are same AS) 

  - path: Cisco-IOS-XR-ipv4-bgp-cfg:/bgp/instance[instance-name=default]/instance-as[as=0]/four-byte-as[as=65000]/vrfs/vrf[vrf-name=CustomerVRF]/vrf-neighbors/vrf-neighbor[neighbor-address=100.100.20.0]/vrf-neighbor-afs/vrf-neighbor-af[af-name=ipv4-unicast]/

    value:

      route-policy-in: AcceptAny

      route-policy-out: AcceptAny

      as-override: true

# To configure Route Target import and export, we will use the Cisco Specific Yang Cisco-IOS-XR-infra-rsi-cfg.yang: 

  # Import Route target

  - path: Cisco-IOS-XR-infra-rsi-cfg:/vrfs/vrf[vrf-name=CustomerVRF]/afs/af[af-name=ipv4][saf-name=unicast][topology-name=default]/Cisco-IOS-XR-ipv4-bgp-cfg:bgp/import-route-targets/route-targets/route-target[type=as]

    value: 

      as-or-four-byte-as:

        as-xx: 0

        as: 1

        as-index: 1

        stitching-rt: 0

  # Export route target

  - path: Cisco-IOS-XR-infra-rsi-cfg:/vrfs/vrf[vrf-name=CustomerVRF]/afs/af[af-name=ipv4][saf-name=unicast][topology-name=default]/Cisco-IOS-XR-ipv4-bgp-cfg:bgp/export-route-targets/route-targets/route-target[type=as]

    value: 

      as-or-four-byte-as:

        as-xx: 0

        as: 1

        as-index: 1

        stitching-rt: 0

# ================================================================================

  # LLDP

  # ================================================================================

  - path: openconfig-lldp:/lldp

    value:


      # Uncomment if you want to enable LLDP globally 

      # config:

      #   enabled: true

      interfaces:

        interface:

          - name: GigabitEthernet0/0/0/1

            config:

              name: GigabitEthernet0/0/0/1

              enabled: true

deletes: 

  # Enable interfaces - works by deleteing Shutdown 

  - Cisco-IOS-XR-ifmgr-cfg:/interface-configurations/interface-configuration[active=act][interface-name=GigabitEthernet0/0/0/1]/shutdown

  - Cisco-IOS-XR-ifmgr-cfg:/interface-configurations/interface-configuration[active=act][interface-name=GigabitEthernet0/0/0/2]/shutdown

Remote Wire Shark

You can capture traffic using SSH Capture in a Containerlab deployment, open WireShark, and the SSH capture :
Remote Server Address : IP of Node running Containerlab
Remote Capture command : $ sudo ip netns exec clab-02_xrv_gnmi-CE1 tcpdump -U -nni eth3 -w -

Template : $ sudo ip netns exec NODE_NAME tcpdump -U -nni INTERFACE_TO_CAPTURE -w -

Useful commands

On Cisco IOS-XR , if you are aconfused of how to implement a gNMI config, you can do it in CLI like you normally do, and then show the gNMI/OpenConfig syntax using below command, enables you to see the config tree : show running-config | xml openconfig

Conatiners used

1- We will use the Praqma image which includes network testing tools pre-packaged :
docker pull praqma/network-multitool
2- Open FRRouting Router for CEs :
docker pull frrouting/frr
3- IOS-XRV Containers, images should be provided by your support.

References

Network Automation : Simulating network topologies with Containers and gNMI

Amr ElHusseiny — Mon, 24 Oct 2022 19:49:30 +0000

Introduction

Like many network engineers, I used to use EVE-NG (or GNS3) to test lab topologies and configurations.Instead of having to control the configuration using the EVE-NG platform, 2 tools available, ContainrLab & vrnetlab enables you to run your most used Network devices in a Docker container platform of your choice, this opens up all the automation tools available for Docker and K8S.

But beware, unlike traditional containers which replaces the resource intensive Virtual Machine with only the needed parts inside a container form, here are only converting those VM to run as VM emulating container, so those containers will use the same amount of resource as the EVE-NG VM if not a little bit more, but you get to optimize the deplyment of these labs.

Simply why would you use this ?
Answer, you Network lab acts like a K8S deployment, so for example, i have multiple topologies templates ready to run with one command if i need to test something, also if i miss it up, with one command I can destroy the lab and deploy another clean instance of the lab.

*What are the supported platforms ? *

Most of the network used platforms are available here, beside all the containerized platforms, also it opens you up to new world of containerizd network devices that cannot be emulatedf with a simple container in EVE-NG (Example Cisco's XRV container if you have access to one).

In this article , we are going to replace the usage of our belove EVE-NG and GNS3, with Containerlab, but to be clear you will find 2 solutions, we are going to explore ContainrLab only :

Solution 1 - VRNetlab : for containerizing the existing "qcow" images we have, and for connectivity between containers, this will be providing the way to package the QCOW into a container form anyways, but the downside with VrnetLab, that it does not provide and easy way to manage the topology, for that ContainerLab implementation of VrnetLab is better in my opinion.
Solution 2 - Containerlab : for Topology management and configuration automation , its based on vrnetlab also, but a modified version, not identical to the standalone Vrnetlab solution . (subject of this article)

Important note : there is 2 versions of vrnetlab, vrnetlab/vrnetlab is not compatiable with ContainerLab, if you want to use ContainerLab, you would have to install hellt/vrnetlab, so the docker image created by one of them cannot be used with the other.

What we plan here is to introduce 2 tools :
1) ContainerLab for provisioning the topology.
2) gNMI (# gRPC Network Management Interface) - OpenConfig project : gPRC to apply configuration templates, replacing Netconf,RestAPI and SSH , supported by multiple carriers and most of the big device manufacturers.

ContainerLab supported platforms (Kinds)

You can find the updated list here

Nokia SR Linux
Nokia SR OS
Arista cEOS
Arista vEOS
Juniper cRPD
Juniper vMX
Juniper vQFX
Cisco XRv9k
Cisco XRv
Cisco CSR1000v
Cisco Nexus 9000v
Cumulus VX
SONiC
Dell FTOS10v
MikroTik RouterOS
Palo Alto PAN
IPInfusion OcNOS
Keysight IXIA-C One
Check Point Cloudguard
Linux bridge
Linux container
Openvswitch bridge

before we start, You can follow the containerlab installation guide here.

Packaging our QCOW into containers

1st you need to provide the image file just like the EVE-NG one, some images has open source or trial versions if your vendor does not provide a testing image.
Here we will try with Cisco's IOS-XR 9K 6.5.1, I suggest you clone the Hellt/VrentLab Github page :
$ git clone https://github.com/hellt/vrnetlab

In here you will find the each supported platform has its directory in which we will place our qcow image,most importantly, each directory conatins a README file instructing you what to nam your image file, and which 'make' command to use so it creates the container image and register ir in your Docker.

For our example for the XR-9K, all you need is to put the qcow image in folder and then run make docker-image.
Now if you check your Docker images list, you will find our router:



$ docker image ls | grep XR
vrnetlab/vr-xrv9k                 6.5.1       be122c04f93a   3 weeks ago     7.07GB

I tag it as latest, so its easier to deploy, however you donot need to that:
$ docker tag vrnetlab/vr-xrv9k:7.2.1 vrnetlab/vr-xrv9k:latest

ContainerLab topology template

You can find the topology files on my Github here :)

Our topology include Cisco's IOS-XR, Junos's VMX and Nokia's SR-Linux to demonstrate the scale:
Primer:

The topology file for Containerlab is a Yaml file devided int 2 parts, the nodes with their naming, docker image "Kind" and initial configuration file if you have that.
The kind is dictated by containerlab, you can find the kind you can put in the Yaml here, make sure you choose the right kind for each platform.
You donot have to include initial/starup configuration file, containerlab by default handles the SSH basic configuration besides other cool configuration liek NetConf and gNMI access.
For my lab, I included configuration files for each node deploying an IS-IS/MPLS backbone as lab template for me to use in the future.
The interfaces names in the topology file is replaced by the Linux 'ethX' naming, you can find the interfaces naming scheme on the kind/platform in the Containrlab site .

Our above topology file looks as follows, easy to read:



# 01_vmx_xrv_lab_topology.clab.yml
name: 01_vmx_xrv_lab_topology

topology:
  nodes:
    PE1:
        kind: vr-vmx
        image: vrnetlab/vr-vmx:21.2R1.10
        startup-config: nodes_configurations/PE1.txt
    P1 :
        kind: vr-vmx
        image: vrnetlab/vr-vmx:21.2R1.10
        startup-config: nodes_configurations/P1.txt
    P2 :
        kind: vr-vmx
        image: vrnetlab/vr-vmx:21.2R1.10
        startup-config: nodes_configurations/P2.txt
    P3 :
        kind: vr-vmx
        image: vrnetlab/vr-vmx:21.2R1.10
        startup-config: nodes_configurations/P3.txt
    P4 :
        kind: vr-xrv9k
        image: vrnetlab/vr-xrv9k:7.2.1
        startup-config: nodes_configurations/P4.txt
    P5 :
        kind: vr-vmx
        image: vrnetlab/vr-vmx:21.2R1.10
        startup-config: nodes_configurations/P5.txt
    P6 :
        kind: vr-xrv9k
        image: vrnetlab/vr-xrv9k:7.2.1
        startup-config: nodes_configurations/P6.txt
    PE2:
        kind: vr-xrv9k
        image: vrnetlab/vr-xrv9k:7.2.1
        startup-config: nodes_configurations/PE2.txt
    PE3:
        kind: srl
        image: ghcr.io/nokia/srlinux:latest

  links:
    - endpoints: ["PE1:eth2", "P1:eth2"]
    - endpoints: ["PE1:eth3", "P5:eth3"]
    - endpoints: ["P1:eth4", "P2:eth4"]
    - endpoints: ["P1:eth5", "P3:eth5"]
    - endpoints: ["P3:eth2", "P4:eth2"]
    - endpoints: ["P3:eth4", "P5:eth4"]
    - endpoints: ["P5:eth2", "P6:eth2"]
    - endpoints: ["P2:eth6", "P4:eth6"]
    - endpoints: ["P2:eth5", "PE2:eth5"]
    - endpoints: ["P6:eth3", "P4:eth3"]
    - endpoints: ["P6:eth4", "PE2:eth4"]
    - endpoints: ["PE3:e1-5", "P1:eth6"]
    - endpoints: ["PE3:e1-6", "P2:eth7"]

To deploy your topology, run :
$ containerlab deploy --topo 01_vmx_xrv_lab_topology.clab.yml

Containerlab provides an output of the topology container management assigned IPs (IPv4/IPv6) and registers it in your hosts file so you can use the hostname instead :



$ cat /etc/hosts
###### CLAB-01_vmx_xrv_lab_topology-START ######
172.20.20.8     clab-01_vmx_xrv_lab_topology-PE2
172.20.20.7     clab-01_vmx_xrv_lab_topology-P4
172.20.20.6     clab-01_vmx_xrv_lab_topology-P1
172.20.20.5     clab-01_vmx_xrv_lab_topology-P5
172.20.20.9     clab-01_vmx_xrv_lab_topology-P6
172.20.20.4     clab-01_vmx_xrv_lab_topology-PE1
172.20.20.10    clab-01_vmx_xrv_lab_topology-P2
172.20.20.2     clab-01_vmx_xrv_lab_topology-PE3
172.20.20.3     clab-01_vmx_xrv_lab_topology-P3
2001:172:20:20::8       clab-01_vmx_xrv_lab_topology-PE2
2001:172:20:20::7       clab-01_vmx_xrv_lab_topology-P4
2001:172:20:20::6       clab-01_vmx_xrv_lab_topology-P1
2001:172:20:20::5       clab-01_vmx_xrv_lab_topology-P5
2001:172:20:20::9       clab-01_vmx_xrv_lab_topology-P6
2001:172:20:20::4       clab-01_vmx_xrv_lab_topology-PE1
2001:172:20:20::a       clab-01_vmx_xrv_lab_topology-P2
2001:172:20:20::2       clab-01_vmx_xrv_lab_topology-PE3
2001:172:20:20::3       clab-01_vmx_xrv_lab_topology-P3
###### CLAB-01_vmx_xrv_lab_topology-END ######

Now, since these are full fledged operating systems, it may take a few minutes to fully boot up, you can follow the progress of the booting process using docker logs like follows :
$ docker logs -f clab-01_vmx_xrv_lab_topology-PE3

After nodes are booted youcan ssh normally like any VM, but easier here since you can use the Hostname (The default username and password can be found on the Containerlab Wiki):



RP/0/RP0/CPU0:PE2\#show interfaces description 
Interface          Status      Protocol    Description
Lo0                up          up          
Nu0                up          up          
Mg0/RP0/CPU0/0     up          up          
Gi0/0/0/0          admin-down  admin-down  
Gi0/0/0/1          admin-down  admin-down  
Gi0/0/0/2          admin-down  admin-down  
Gi0/0/0/3          up          up          description to_P6_Link_11
Gi0/0/0/4          up          up          description to_P2_Link_10

You can see from the output, that some interfaces are already configured, as it booted up and deployed my configuration file.

Some useful operations commands

To show current topology running nodes :



$ containerlab inspect --topo 01_vmx_xrv_lab_topology.clab.yml
+---+----------------------------------+--------------+------------------------------+----------+---------+-----------------+----------------------+
| # |               Name               | Container ID |            Image             |   Kind   |  State  |  IPv4 Address   |     IPv6 Address     |
+---+----------------------------------+--------------+------------------------------+----------+---------+-----------------+----------------------+
| 1 | clab-01_vmx_xrv_lab_topology-P1  | fcbc335b26b2 | vrnetlab/vr-vmx:21.2R1.10    | vr-vmx   | running | 172.20.20.6/24  | 2001:172:20:20::6/64 |
| 2 | clab-01_vmx_xrv_lab_topology-P2  | 2c9ca5de1124 | vrnetlab/vr-vmx:21.2R1.10    | vr-vmx   | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 3 | clab-01_vmx_xrv_lab_topology-P3  | 2ad9bc2c91e4 | vrnetlab/vr-vmx:21.2R1.10    | vr-vmx   | running | 172.20.20.3/24  | 2001:172:20:20::3/64 |
| 4 | clab-01_vmx_xrv_lab_topology-P4  | de4c72b64214 | vrnetlab/vr-xrv9k:7.2.1      | vr-xrv9k | running | 172.20.20.7/24  | 2001:172:20:20::7/64 |
| 5 | clab-01_vmx_xrv_lab_topology-P5  | aaa1ae3a4887 | vrnetlab/vr-vmx:21.2R1.10    | vr-vmx   | running | 172.20.20.5/24  | 2001:172:20:20::5/64 |
| 6 | clab-01_vmx_xrv_lab_topology-P6  | e3ad22362111 | vrnetlab/vr-xrv9k:7.2.1      | vr-xrv9k | running | 172.20.20.9/24  | 2001:172:20:20::9/64 |
| 7 | clab-01_vmx_xrv_lab_topology-PE1 | 6d0de04511d9 | vrnetlab/vr-vmx:21.2R1.10    | vr-vmx   | running | 172.20.20.4/24  | 2001:172:20:20::4/64 |
| 8 | clab-01_vmx_xrv_lab_topology-PE2 | 0c78217813d3 | vrnetlab/vr-xrv9k:7.2.1      | vr-xrv9k | running | 172.20.20.8/24  | 2001:172:20:20::8/64 |
| 9 | clab-01_vmx_xrv_lab_topology-PE3 | abdcdb9e2b08 | ghcr.io/nokia/srlinux:latest | srl      | running | 172.20.20.2/24  | 2001:172:20:20::2/64 |
+---+----------------------------------+--------------+------------------------------+----------+---------+-----------------+----------------------+

Or you can show all the current running Containerlab nodes on host, even if they are from diffirent topologies:
$ containerlab inspect --all

Finally, you can destroy the topology or all the running instances in one command:
$ containerlab destroy --topo 01_vmx_xrv_lab_topology.clab.yml
$ containerlab destroy --all

More labs

You can find more example labs already createdt by the Containerlab creator, which is very useful here

Using gNMI & OpenConfig for Automation

Introduction

Starting with SSH movint to SNMP then moving to Netconf then RESTful APIs, there has been multiple iterations for config management and telemtry in the network space, due to the vast variety of networking vendors, form factors and specilaization, adoption of new technologies for management and telemetry streaming has been very hard, thankfully a new group of industry leaders has joined the OpenConfig initiative which is trying to create a standard for interoperabilty.

the gRPC protocol (initially developed by Google) is the next evolution of Rest and Netconf, it runs on top of HTTP2, making it TCP reliable and allows the ability for Streaming Telemetry, which makes it instrumental in replacing SNMP and then adopted industry wide, It is targeted towards the expansion of Microservices, it enables you to use alomost any of the well known languages to build each part of your application whether the same language or not, and then using gRPC just like REST, you are able to communicate between these microservices in mutually understandable way.

One of the important diffireneces betwee gRPC and REST is the introducation of data serilaization using ProtoBuff (Protocol Buffer) instead of JSON which is mostly a little bigger in size and more porccessor intensive, unlike JSON though ProtoBuff is not humanlly readable since its represented in Binary form but very efficient banddwidth and processing wise.

gNMI (gRPC Network Management Interface) uses gRPC protocol but specilaizes in the Network devices domain , you can refer to Cisco's page for more detail about the gNMI protocol, as for OpenConfig, vendors adhere to some model driven standards to implement a similar Yang models so you can get/set config , subscribe to telemetry in a standard Yang representation without having to adapt your code to each new vendor you have.

Limitations of gNMI and OpenConfig

While gNMI is fully implemented for most of the participating vendors, with some limitations for example with Juniper's Junos there is still some issue with Get Config request, generally you should be able to use gNMI as your tool.

Most importantly jsut like Netconf and SNMP MIBs, the OpenConfig standards does not cover all topics and features, so you may have to use the vendor's specific Yang models instead of the OpenConfig ones while using those specfic features.

Be vigilant to follow up with the vendor's implementation as they share it publicly, here are some Github pages which you can source your , one of the most adhering vendors is Nokia, their implementation is impcable, you can start to learn from there .

Cisco's IOS-XR/NexusOS/IOS-XE/NCS supported Yang models https://github.com/YangModels/yang/tree/main/vendor/cisco
Juniper's Junos MX/SRX/PTX Yang Models : https://github.com/Juniper/telemetry
Nokia's SR-Linux Yang Models : https://github.com/nokia/srlinux-yang-models

Now you have download some Yang models, that reflect configuration hirarchy, now how can you read those Yang models, lets have a look at some tools to help out with that.

pyang

PYANG is a program, that can be used to validate YANG models against RFCs and more importantly generate a tree representation of a Yanf module.

Lets say, you want to see how are the configuration of interfaces is impelemented in Cisco's IOS-XR OpenConfig compliant Yang model.

1st install pyang:
$ pip install pyang
Now, you can either get the OpenConfig models from the OpenConfig repo which should work properly with complying vendors, or the way i prefer is to get it from the vendor's page, as you will find additional Yang models specific to each vendor, here we will get it from Cisco's Git.
$ git clone https://github.com/YangModels/yang.git cisco_yang_models $ cd cisco_yang_models/yang/vendor/cisco/xr/651/
Lets have a look onto a Yang tree using pyang , the rw referes to read/write , meaning these are configurable , the '[name]' means this is the key for this list, we will use keys later to configure specfic interface or to get its stats :



$ pyang openconfig-interfaces.yang -f tree
module: openconfig-interfaces
  +--rw interfaces
     +--rw interface* [name]
        +--rw name             -> ../config/name
        +--rw config
        |  +--rw type           identityref
        |  +--rw mtu?           uint16
        |  +--rw name?          string
        |  +--rw description?   string
        |  +--rw enabled?       boolean

Like I said, some Yang models are still vendor specific like the Segment Routing module from Cisco for example:



$ pyang Cisco-IOS-XR-segment-routing-ms-cfg.yang -f tree 
module: Cisco-IOS-XR-segment-routing-ms-cfg
  +--rw sr
     +--rw local-block!
     |  +--rw lower-bound    uint32
     |  +--rw upper-bound    uint32
     +--rw mappings
     |  +--rw mapping* [af ip mask]
     |     +--rw af               Srms-address-family
     |     +--rw ip               inet:ip-address-no-zone
     |     +--rw mask             uint32
     |     +--rw sid-start?       uint32
     |     +--rw sid-range?       uint32
     |     +--rw flag-attached?   Srms-mi-flag
     +--rw adjacency-sid
     |  +--rw interfaces
     |     +--rw interface* [interface]
     |        +--rw address-families
     |        |  +--rw address-family* [address-family]
     |        |     +--rw next-hops
     |        |     |  +--rw next-hop* [ip-addr]
     |        |     |     +--rw l2-adjacency-sid
     |        |     |     |  +--rw sid-type?       Sid-type-list
     |        |     |     |  +--rw absolute-sid?   uint32
     |        |     |     |  +--rw index-sid?      uint32
     |        |     |     |  +--rw srlb            Srlb-string
     |        |     |     +--rw ip-addr             inet:ip-address-no-zone
     |        |     +--rw address-family    Srms-address-family
     |        +--rw interface           xr:Interface-name
     +--rw global-block!
     |  +--rw lower-bound    uint32
     |  +--rw upper-bound    uint32
     +--rw enable?          empty

gNMIc

gNMIc is a tool developed initiatlly by Nokia and then donated to OpenConfig, providing a platform to implement gNMI management, including a very useful CLI tool.

Lets see what we can do with, 1st like pyang , we can list the available configuration tree, but this time in a more useful Path Value format, this shows you the Configuration Paths you can use in the gNMI call



gnmic path openconfig-interfaces --file openconfig-interfaces.yang --config-only
/interfaces/interface[name=*]/config/description
/interfaces/interface[name=*]/config/enabled
/interfaces/interface[name=*]/config/mtu
/interfaces/interface[name=*]/config/name
/interfaces/interface[name=*]/config/type
/interfaces/interface[name=*]/description
/interfaces/interface[name=*]/enabled
/interfaces/interface[name=*]/hold-time/config/down
/interfaces/interface[name=*]/hold-time/config/up
/interfaces/interface[name=*]/link-up-down-trap-enable
/interfaces/interface[name=*]/name
/interfaces/interface[name=*]/name
/interfaces/interface[name=*]/subinterfaces/subinterface[index=*]/config/description
/interfaces/interface[name=*]/subinterfaces/subinterface[index=*]/config/enabled
/interfaces/interface[name=*]/subinterfaces/subinterface[index=*]/config/index
/interfaces/interface[name=*]/subinterfaces/subinterface[index=*]/config/name
/interfaces/interface[name=*]/subinterfaces/subinterface[index=*]/index
/interfaces/interface[name=*]/type

Now lets put it to use with our Lab topology, we can get the configuration of one of PE2 interfaces like follows :
Get Example :



$ gnmic -a clab-01_vmx_xrv_lab_topology-PE2  --insecure -u admin -p admin -e json_ietf  get --path 'openconfig-interfaces:/interfaces/interface[name="Loopback0"]'
[
  {
    "source": "clab-01_vmx_xrv_lab_topology-PE2",
    "timestamp": 1666636579876879221,
    "time": "2022-10-24T18:36:19.876879221Z",
    "updates": [
      {
        "Path": "openconfig-interfaces:interfaces/interface[name=\"Loopback0\"]",
        "values": {
          "interfaces/interface": [
            {
              "config": {
                "enabled": true,
                "name": "Loopback0",
                "type": "iana-if-type:softwareLoopback"
              },
              "name": "Loopback0",
              "state": {
                "admin-status": "UP",
                "enabled": true,
                "ifindex": 11,
                "last-change": 33469,
                "mtu": 1500,
                "name": "Loopback0",
                "oper-status": "UP",
                "type": "iana-if-type:softwareLoopback"
              },
              "subinterfaces": {
                "subinterface": [
                  {
                    "index": 0,
                    "openconfig-if-ip:ipv4": {
                      "addresses": {
                        "address": [
                          {
                            "config": {
                              "ip": "100.64.0.22",
                              "prefix-length": 32
                            },
                            "ip": "100.64.0.22",
                            "state": {
                              "ip": "100.64.0.22",
                              "origin": "STATIC",
                              "prefix-length": 32
                            }
                          }
                        ]
                      }
                    }
                  }
                ]
              }
            }
          ]
        }
      }
    ]
  }
]

Update Example : for update, currently i cannot use the OpenConfig interfaces module to do the update for some reason, so instead, i will use the Cisco provided interfaces module, I will update the Loopback0 description :



$gnmic -a clab-01_vmx_xrv_lab_topology-PE2  --insecure -u admin -p admin -e json_ietf  set --update-path 'Cisco-IOS-XR-ifmgr-cfg://interface-configurations/interface-configuration[active="act"][interface-name="Loopback0"]/description' --update-value "management_interface"
{
  "source": "clab-01_vmx_xrv_lab_topology-PE2",
  "timestamp": 1666638282750917788,
  "time": "2022-10-24T19:04:42.750917788Z",
  "results": [
    {
      "operation": "UPDATE",
      "path": "Cisco-IOS-XR-ifmgr-cfg:interface-configurations/interface-configuration[interface-name=\"Loopback0\"][active=\"act\"]/description"
    }
  ]
}

lets confirm change on the device , see it changed:



RP/0/RP0/CPU0:PE2#show interfaces loopback 0 description 
Interface          Status      Protocol    Description
/--------------------------------------------------------------------------------
Lo0                up          up          management_interface

you can try it with more complex modules like for example the VRF module from Cisco 'Cisco-IOS-XR-ip-iarm-vrf-cfg', with the help of pyang tree command you can explare the possible change.

Next article we will explore the SNMP replacing gNMI Subscription telemetry.

Useful commands :

To show all the paths avilable in a Yang model ( gnmic - bash ) $ gnmic path openconfig-interfaces --file openconfig-interfaces.yang
To only see the paths available to be configured gnmic path openconfig-interfaces --file openconfig-interfaces.yang --config-only
To run gNMI requests using template inventory "config.yaml" containing the targets, and configuration template "nodes_configuration.yaml" $ gnmic --config config.yaml set --request-file nodes_configuration.yaml --log
To show the tree structure of the Yang model ( pyang - python ) $ pyang Cisco-IOS-XR-ifmgr-cfg.yang -f tree
Enable External LAN access to Containerlab containers sudo iptables -vnL DOCKER-USER

References

RPKI , securing BGP Infrastructure - with Lab

Amr ElHusseiny — Tue, 13 Sep 2022 07:51:56 +0000

Introduction
- What & Why RPKI ?
- Signing ROAs
  - what is an ROA ?
- How you deploy ?
*RPKI-RTR architecture *
- PDUs Structure (rpki-rtr-pdu)
- Protocol workings
  - Serial Query/Cache Response
  - Serial Notify
  - End of data
  - Reset Query
Applying RPKI [Lab]
- 1) Setup a Validator
- 2) Discarding BGP invalids / configuring routers
- 3) Validating RPKI operations
MANRS (Mutually Agreed Norms for Routing Security)

Introduction

What & Why RPKI (Resource Public Key Infrastructure)?

BGP is the routing protocol of the Internet backone, but it always lacked a way to validate recieved routes, this opened the protocol up tp some nasty attack of which one of the more popular ones is Route Hijacking in which a peer would falsely or mistakingly advertise some routes as if it is the originator, you can read more about an example of this case in Taiwan's TWNIC hijack in 2019, intentional attack could have resulted in either blackholing the traffic, or driving traffic towards a sniffing device to spy.

In order to overcome this, and RFC was created RFC-6810 and a work group was spun up to push for the implementation of this. feature represented by MANRS (Mutually Agreed Norms for Routing Security).

What happens that the BGP running device validates the recieved BGP route againt the RPKI local database to confirm the validty of the originating AS.

RPKI is used for Origin Validation and not AS-Path validation , RPKI validation process only looks at the OriginAS in the Route NLRI, however it does not look in AS Path, so you may have a hijacker start a malicious router with your AS number behind his own AS and use that to advertise your prefix, or AS-Path, you should be doing Prefix filtering on your boundaries anyways.

Signing ROAs

1st you need to understand that there is 5 Certificate Authorities in conjunction with the RIRs (Regional Internet Registry) charged with creating and validating the carriers certificates, you should register ROAs with your RIR, using your already created service provider account.

side note - Certificate providers are AFRINIC, APNIC, RIPE NCC, LACNIC, ARIN.

what is an ROA ?

The Route of Origin Authorization simply consists of a Prefix and its Origin, indicating which AS (Autonomous System) is authorised to advertise this Prefix.
Multiple ROAs can exist for the same Prefix, ROAs can overlap.

I can better show you how an ROA looks like from a captured RPKI Route Update :

in the above snapshot, means you can advertise any network in between the /22 and /24 subnet mask, so its not only allowing you advertise one subnet, but a range of subnet masks.

Cool way to search for the registered prefixes on NLNOG's IRR Exp , this enables you to see which IRR is a particular ROA is registered with:

How you deploy ?

Simply you will need to do the following, we will go more in detail later:
1) Create ROAs (Route of Origin Authorization).
2) Setting up Validators (Validation Cache Servers).
3) Discarding BGP invalids on boundry routers .

RPKI-RTR architecture

RPKI-RTR is the protocol used in between the validator and the Client routers, the RFCs states the structure to deploy the RPKI service as per the following driagram :
Note : Local Cache = Validator

PDUs Structure (rpki-rtr-pdu)

The update header includes the following :

Protcol Version : currently version 0
PDU Type : IPv4 for example
Serial Number : Indicates the last update number recieved from the upstream Cache or Global RPKI server, so if the downstream client router sends his last PDU Serial number to the upstream Cache , the Cache will send the client router all PDUs with greater Serial number which correspinds to the the updates that this client router did not recieve yet .
Session ID : Used in conjunction with the Serial number to identify these Serial Numbers are still attached to the same session that is established with the upstream server.
Length : length of the whole PDU
Flags : 1 for Announcement - 0 for Withdraw
Prefix Length : Shortest Prefix (Min)
Max Length : Longest Prefix (Max)
Prefix : The IP Portion
AS : 32-bit format AS number (Autonomous Number)

Note : IPx PDU uniqueness is identified by the set of [Prefix, Len, Max-Len,ASN]

Protocol workings

RPKI uses the port rpki-rtr TCP 323, also cache to router communication is recommeneded to use TCP-A0 (you can check RFC-5925) for authentication instead of TCP-MD5.

Serial Query/Cache Response

A session is kept up between either the RPKI Cache and the Global RPKI server , or between the downstream RPKI client router and the RPKI Cache.
The router periodically sends the Cache server the last (greatest) PDU Serial number , the cache responds by sending all the updates with a higher Seral number, then the client router updates its current serial number to the last PDU recieved.
To make sure that the data is up to date a Keepalive like interval of 1 hour, at which the Client router should send a Serial Query to make sure his updates are up to date.

Serial Notify

When the RPKI Cache recieves updates, it send a Notification to its clients, saying this is a good time to Poll for new data, Client routers must be the one initiating the requests for the update .

End of data

If the Cache recieves a query and finds that there is no updates to send to the Client router, instead it sends an End of data PDU

Reset Query

Client router requests the whole current updates database from the Cache server, this message can be sent in the rest or the start of the connection so the Client can recieve a full table f updates from the caching server, initiation looks as follows .

Applying RPKI

We are using Ubuntu with Routinator which is a RIPE developed Route Validator, and 2 IOS-XE routers, Router_1 is RPKI Client, and Router_2 is the eBGP peer that will send the invalid subnet 1.0.0.0/24 .

After ROA is created, it may take to 10 minutes to show up in the Registries databases and would be able to validate.

1) Setup a Validator

There is multiple validators available on the market , we will be using Routinator ,a validator developed by RIPE.
Remember RPKI Validators downloads the whole RPKI Repository from the RIRs databases, acting as cache afterwards, keeping its database updated by queriing the RIRs periodically .

There are multiple Validtor softwares available other than Routinator if you prefer such as OpenBSD's RPKI-Client , Cloudflare's OctoRPKI , or FORT's Validator.

For now we are going to use Routinator, but feel free to use any of these packages.

steps

Update and install gcc C compiler, then installing Rust as Routinator is written in Rust, after that we run it .

$ sudo apt update && sudo apt upgrade
$ sudo apt-get gcc
$ curl https://sh.rustup.rs -sSf | sh 
$ source $HOME/.cargo/env 
$ cargo install routinator

All RIRs repos are by default open to be queried with no issue, but for ARIN, you have to acknowledge you have read the terms and accept, to accept the ARIN's aggreement :
$ routinator init --accept-arin-rpa
Following command will show and print out the whole Global IP Address allocation (ROAs) , it could some time , i would advise to run it in the background :
$ routinator -v vrps
To have the Routinator(Validator/RPKI Cache) start listening to Client's/Router's request, start the service with the following command, the default port for RPKI should be 323, but to avoid running the server as root, we will be using port 3323 instead :
$ routinator server --rtr 192.168.1.143:3323 --refresh=900
Following how your newy installed validator communication looks like

Allow List

You can create a whitelist locally in your validator, to add exceptions for some subnets that you donot need validated.

2) Discarding BGP invalids / configuring routers

On a Cisco IOS-XE router, its as simple as running the following command under the "router bgp xxxxx" config

My Authenticator server IP is 192,168.1.143 and port 3323 is the listening port : Router(config-router)#bgp rpki server tcp 192.168.1.143 port 3323 refresh 900 Now on the Cisco router I am connected to the validator server and able to see the established session

Router#show ip bgp rpki servers 
BGP SOVC neighbor is 192.168.1.143/3323 connected to port 3323
Flags 64, Refresh time is 900, Serial number is 0, Session ID is 30149
InQ has 0 messages, OutQ has 0 messages, formatted msg 1
Session IO flags 3, Session flags 4008
 Neighbor Statistics:
  Prefixes 299821
  Connection attempts: 1
  Connection failures: 0
  Errors sent: 0
  Errors received: 0

and you can see the whole ROA table

Router(config-router)#do show ip bgp rpki tab
228256 BGP sovc network entries using 36520960 bytes of memory
250017 BGP sovc record entries using 8000544 bytes of memory
Network              Maxlen  Origin-AS  Source  Neighbor
1.0.0.0/24           24      13335      0       192.168.1.143/3323
1.0.4.0/24           24      38803      0       192.168.1.143/3323
1.0.4.0/22           22      38803      0       192.168.1.143/3323
1.0.5.0/24           24      38803      0       192.168.1.143/3323
1.0.6.0/24           24      38803      0       192.168.1.143/3323
.......

Note that this table is not your routing table, its for validation not for route reflection.
Now to further have a look about the PDUs , i setup a wireshark capture, after that, you need to go to the capture stream of the port 3323 and click decode as/current/RPKI-Router-Protocol like the following screenshot

You can also use the RPKI validation in your router's route policy's (Cisco, Juniper and so on), for instance you can set a preference for the validated routes, you donot need to reject the non-validated routes, you can just set a worse preference for them.

3) Validating RPKI operations

eBGP configured between Router_1 (RPKI Client) and Router_2 (Hijacking router) , and network 1.0.0.0/24 is advertised from Router_2 to Router_1, when we inspect Router_1's routing table we see the subnet not present, meaning RPKI is working.
To see the reason for rejecting the route on Router_1

Router_1#show bgp 1.0.0.0
BGP routing table entry for 1.0.0.0/24, version 0
Paths: (1 available, no best path)
  Not advertised to any peer
  65004
    10.0.0.2 from 10.0.0.2 (10.0.0.2)
      Origin IGP, metric 0, localpref 100, valid, external
     path 7FA8A922B2E8 RPKI State invalid  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
      rx pathid: 0, tx pathid: 0

MANRS

(Mutually Agreed Norms for Routing Security)
MANRS is a group of operators initiative pushing for route security and security actions, they publish an Actions whitepaper for service providers how to participate and which actions is compulsory and which actions are options.
You can find detailed explanation in their publication on manrs.org.

Compulsory Actions to join

1) Prevent probagation of incorrect routing information : mainly confirm their peer service provider actually holds that IP Prefix and AS number.
2) Correct up to date NOC Contact for carriers correspinding to prefixes/ASs should be registered and kept current with Regional and/or National regisry (RIR & NIR), can be registered with the PeeringDB (PeeringDB Network Operators HowTo).
3) Carriers must publicly document their owned subnets/ASs and also which prefixes/ASs they provide Transit services for through their RPSL with IRRs, or RIRs, and NIR databases, also recommended to create ROAs (Route of Origin Authorizations) foe the list of prefixes will be advertised by the carrier.
4) Prevent Spoofed traffic by using source address filteration or a tool like uRPF.

References

Extending RFC1918 with RFC-6598 (100.64.0.0/10)

Amr ElHusseiny — Sat, 10 Sep 2022 19:18:54 +0000

introduction

We are all familiar with the RFC-1918 AKA Private IP Spaces (10.x.x.x , 172.16.x.x , 192.168.x.x), called private IP addresses because they can be duplicated by any user as long as the duplicated IP assigned devices does not meet in same L3 network.

But due to the exhaustion of the Public IPv4 space , IANA has assigned a second address space like RFC-1918 , this time though RFC-6598 and its called the Shared Address space since 2012 , while it was originaly intended to be used for CGNAT (Carrier Grade Natting), IANA stated in the standard, that it also can be used in the same manner as RFC-1918 addresses , quoting from RFC-6598:

Use of Shared CGN Space Shared Address Space is IPv4 address space designated for Service Provider use with the purpose of facilitating CGN deployment. Also, Shared Address Space can be used as additional non-globally routable space on routing equipment that is able to do address translation across router interfaces when the addresses are identical on two different interfaces.

The newly assigned address space is the 100.64.0.0/10, you may be familiar with, for me was 1st exposed to the usage of this address space in Datacenters's devices loopback interfaces, wherther network devices or baremetal servers, thats why I went on a journy to do research on the usage of this IP Space .

to pivot a little bit for you to understand what is CGNAT before we continue.

CGNAT (Carrier Grade NAT)

Described very simply in the following diagram :

Before : each home CPE was assigned a Public IP address to use to reach internet directly .
After : Home CPEs are assigned RFC-6589 IP Address (100.64.0.0/10) that will be Natted to Public IP ranges on the edge of your ISP , still the NATing is one to one , but you will only need a Public IP address if you want to reach external resources , as long as you stay in your carrier's network you will beusing your RFC-6598 IP.

You may ask why not just use RFC-1918 IPs for home CPE, while its possible, it showed to cause operational issue with the carriers.

One of the considerations that carriers should start doing , is including the 100.64.0.0/10 in their incoming route filter list from other carriers as a security mediation , just like all carriers do with the RFC-1918 ranges , quoting the RFC.

routing information about Shared Address Space networks MUST NOT be propagated across Service Provider boundaries. Service Providers MUST filter incoming advertisements regarding Shared Address Space.

back to our main topic

Special Use IP Addresses

Since we are having a look at one of the new address spaces, lets take this opportunity to get to know some of the unfamiliar special use IP ranges stated by IANA in RFC-5735

This is to extend your knowledge of the RFCs specs , but not all Networks Engineers adhere to those testing IP ranges, we mostly see the compliance in the Operational spaces only .

Address Space	Usage
169.254.0.0/16	This is the "link local" block. As described in [RFC3927], it is allocated for communication between hosts on a single link. Hosts obtain these addresses by auto-configuration, such as when a DHCP server cannot be found.
192.0.2.0/24 , 198.51.100.0/24,	This block is assigned as "TEST-NET" for use in documentation and example code. It is often used in conjunction with domain names example.com or example.net in vendor and protocol documentation
192.88.99.0/24	This block is allocated for use as 6to4 relay anycast addresses
198.18.0.0/15	This block has been allocated for use in benchmark tests of network interconnect devices, this range was assigned to minimize the chance of conflict in case a testing device were to be accidentally connected to part of the Internet.

Q&A

Why shouldn't use Shared Address Space range in your Enterprise network ? , in the renound Ivan Pepelnjak article , he argues that you should never use this IP address space (100.64.0.0/10) as its not considered Private and it will result in issues if you are using that address space in your enterprise sites at the same time that your Servic Provider is using CGNAT as he would probably using IPs from the same space on his CPE external interface .
When you should use Shared Address Space IPs in your enterprise ? If you ran out of IPv4 private addresses and you are unable to migrate to IPv6 , DONOT use them to interconnect with your service provider and DONOT advertise them your service provider.
Where is it used ? some Datacenter providers and clouds use these addresses in the underlay as they are less likely to conflict with any overlay customer's address which probably is in the RFC-1918 ranges.
Why Cloud providers are pivoting to use the Shared Address Space in their Data centers? At 1st I thought when we used it in our Datacenter underlay that we would be the only ones doing that, but it seems to be an industry wide direction in the Cloud providers specifically, found that also VMWare does the same , you can check VMware SDDC
How many addresses available in the Private/Shared address ranges ? | Address Range | Number of IPv4 addresses | |----------------|--------------------------| | 10.0.0.0/8 | 16 Million addresses | | 172.16.0.0/12 | 1 Million addresses | | 192.168.0.0/16 | 65536 Addresses | | 100.64.0.0/10 | |

References

Following are useful RFCs to understand the reason behind the push and the issues faced by carriers implementing CGNAT

Automating DrawIO network topology using Python

Amr ElHusseiny — Tue, 06 Sep 2022 19:42:51 +0000

Intro

In this blog , we are going to show how to use the draw_network_plot python library to generate a DrawIO Network topology easily instead of manually drawing them yourself .

DrawIO is widely used as a free alternative to Microsoft's Visio to draw network topologies.

I will be using Netmiko and CDP to acquire the data needed for the plot for the devices themselves , but this part won't be the focus of the article .

Focus of the article is how to use the "drawio_network_plot" library" to generate the DrawIO file .

Library setup

Make sure to have Python 3.7 and above , then install the library using :
$ pip install drawio_network_plot

Lab setup

Lab is done on EVE-NG using Cisco's 7200VXR image to create the topology , the whole topology is L3 with Loopback so the Python CentOS server would be able to ssh to each device .

EVE-NG Topology Diagram (Non-generated):

Configuration and Script

You can find both the below script and all the EVE-NG Cisco devices configuration in the following Github Link :
drawio_network_plot/examples/cisco_gather_cdp_and_plot



from netmiko import ConnectHandler
import re 
from drawio_network_plot.drawio_network_plot import NetPlot

# ----------------- Getting live CDP Neigbors to gather links endpoints ---------------------------
def retieve_lldp_neigbor_hostname(devices_list):
    list_of_cdp_neighborship = []
    for device in devices_list:
        device_dictionary = {
                            'device_type': 'cisco_ios',  
                            'host': device['ip_address'],
                            'username': 'automation',
                            'password' : '1234567',
                        }
        net_connect = ConnectHandler(**device_dictionary)
        output = net_connect.send_command("show cdp neighbor")
        lines_list = output.splitlines()
        for line in lines_list:
            try:
                # regex to search the name of the device before the domain ID ".default" , then removing the word ".default" from the string
                link = {
                            'sourceNodeID' : device['nodeName'],
                            'destinationNodeID' : re.search('\S+\.default',line).group().replace('.default','') 
                            }
                # checking for duplication before adding new link : 
                if {'sourceNodeID':link['destinationNodeID'],'destinationNodeID':link['sourceNodeID']} not in list_of_cdp_neighborship:
                    list_of_cdp_neighborship.append(link)
            except:
                continue
        net_connect.disconnect()
    return list_of_cdp_neighborship

def main():
    # Lab Devices , must have the device type for the plotting library to work 
    devices = [
            {'nodeName':'Router_1','ip_address':'100.0.1.1','nodeType':'router','nodeDescription':'NA'},
            {'nodeName':'Router_2','ip_address':'100.0.1.2','nodeType':'router','nodeDescription':'NA'},
            {'nodeName':'Spine_1','ip_address':'100.0.2.1','nodeType':'l3_switch','nodeDescription':'NA'},
            {'nodeName':'Spine_2','ip_address':'100.0.2.2','nodeType':'l3_switch','nodeDescription':'NA'},
            {'nodeName':'Leaf_1','ip_address':'100.0.3.1','nodeType':'l2_switch','nodeDescription':'NA'},
            {'nodeName':'Leaf_2','ip_address':'100.0.3.2','nodeType':'l2_switch','nodeDescription':'NA'},
            {'nodeName':'Leaf_3','ip_address':'100.0.3.3','nodeType':'l2_switch','nodeDescription':'NA'}
            ]
    # Getting list of links for each device
    list_of_cdp_neighborship = retieve_lldp_neigbor_hostname(devices)
    for peering in list_of_cdp_neighborship:
        print(peering)

    # ------------------------------------------------------------------------------------------------------
    # ------------------------------------------------------------------------------------------------------
    # ----------------- Main Part : using library to generate XML DrawIIO format ---------------------------
    # Using the Plot library 
    x = NetPlot()
    x.addNodeList(devices)
    x.addLinkList(list_of_cdp_neighborship)
    print(x.display_xml())
    # ------------------------------------------------------------------------------------------------------
    # ------------------------------------------------------------------------------------------------------
    # ------------------------------------------------------------------------------------------------------

if __name__ == "__main__":
    main()

Library Options

Remember , you can gather the data however you like , this was just a simple demonstration of what you can do , main code to remember is the plotting part :



# please adhere to the naming scheme in the variables 
device_list = [{'nodeName' : 'TOR_1','nodeType' : 'l2_switch','nodeDescription' : 'Leaf Switch 01'}]
x = NetPlot()
# **IMPORTANT NOTE :**---> Make sure that the sourceNode in the connection is the higher level device and that connections are not replicated , this way when you use the DrawIO automatic layout , it would create the diagram hierarchy the correct way 
connections_list = [{'sourceNodeID' : 'Router_1','destinationNodeID' : 'Core_switch_1'}]
    # Adding using list all at once
    x.addNodeList(device_list)
    x.addLinkList(connections_list)
    # Adding node by node and link by link
    x.addNode(nodeName='Router_18',nodeType='router')
    x.addLink('Router_17','Router_18')

    # --- Output ---
    # You can print the XML to the Stdout 
    print(x.display_xml())
    # Or You can also directly generate an XML file using the built in function :
    # x.exportXML('examples/output.xml')

Generated output will be collapsed in one point , you will have to choose the Layout you would like in DrawIO after opening the file like showed as follows :

Generated Topology

1- Open generated XML file in DrawIO

2- go to Arrange/Layout/Vertical Tree

3- Final Result

For any comment on the pip package , feature addition or any comment , please share your suggestions as this is my 1st PIP library , and divinely its not perfect .

Thanks ...

Understanding networks ASICS - Part 1

Amr ElHusseiny — Sun, 04 Sep 2022 21:07:54 +0000

Intro

For people working in the planning portion of networks , they will face important decision for whether to expand on an exiting chassis they already are running or whether they are going with a new vendor or model, if this is your first time to do that , what are the questions you need to ask yourself or your vendor ?

Depending on the network these chassis are going to be deployed , you need to ask the following :

• Routing vs Switching - Are you going to run a routing L3 protocol , or you just need a Web scale L2 switching ?
• Feature Rich vs High Speed - Do you need feature rich such for enterprise edge , or datacenter TOR or backbone service router for example or very fast and lean spine or backbone switching device ?
• Deep buffer vs Shallow buffer - Does your network benefit from a Deeper off chip sizable buffer or Shallower faster on-chip buffer ?
• Programmable vs Fixed - Are you intending to program the chip yourself/vendor will you just use the chip's features as hardware allows ?
• Advanced vs Basic traffic management - Are you going to implement a legacy basic routing control over traffic , or will you start using intent based controller based routing policies ?
• Fixed box vs Modular chassis - Do you need a 1RU box or multi unit expandable modifiable chassis ?

There is a fierce competition in the Ethernet silicon area in different directions, there is the competition between Commercial general purpose silicon and the traditional Merchant silicon providers in the cost , throughput and features areas .

It used to be feature rich against high throughput silicons , but its no longer that simple since the cost of chips now mostly depend on the size of the silicon die (portion of the processor holding the transistors), silicon fabricators already set some standard sizes that processor designer have to adhere to .

Silicon chips used to face a problem with yields because processors comprised of a single die which became more costly each time you increase its size, but lately there has been a direction to break that single piece of silicon to multiple "Chiplets" with cores on a multiple chiplets and IO maybe on another chiplet , these chiplets even used different transistor sizes to optimize the cost against the feature list , so for instance you may have your CPU core made in 7nm process node you can have the IO chiplet made in the 14nm process node size to make it more cost effective , this approach was pioneered by AMD in its EPIC datacenter CPU family , and in the network space it started to appear with Intel's Tofino 2 family bundling Tofino 2 7nm Cores with , read more about it in Intel Tofino 2 pioneering chiplets design

ASICS Pipelines

Basically ASICs works in a bunch of Pipelines , for instance in a Network ASIC for example , a pipline dictates the way traffic is being processed from the moment it ingresses through a port to egressing through another port :

Cisco 3100 Architecture

The parts of the pipeline represents the feature sets , so when a TAC support advises , that a feature is not supported in Hardware , it means that its not implemented in the ASIC's Pipeline, to give an example some chassis does not support VXLAN forwarding, meaning the ASIC's pipeline does not feature a part dedicated for VXLAN encapsulation and decapsulation imprinted on the hardware .

Sometimes for vendor instead of redesigning a new ASICs pipline to support a new feature , they would reprogram an existing ASIC and do what is called a recirculation , so you want to add support for VXLAN for instance , you would have to go 1st time through the Pipeline to decapsulate the VXLAN header 1st , then recirculate the traffic second time into the same pipeline but this time for L3 lookup , as these actions was not designed on the ASIC to be supported at the same time , some behavior we used to see with GRE and MPLS , you had to choose only one if you want to pass the pipeline once , if you wanted both then you have to pass the pipeline twice , this of course doubles the latency .

So vendors would go in a Tic Toc cycle of adding feature thus creating a feature rich platform (like Broadcom's Trident family) , the tic , then optimizing the chip design to process those new features on a faster hardware level (like Broadcom's Tomahawk family) .

Programmability

Some ASICS , you would have access to the SDK (Example : P4 , OpenFlow or even hardware level Assembly SDK), some are fixed and some are in-between , example for programable networking targeted chips like the Intel Tofino family or Cisco's latest Silicon One chips that supports P4 programing language.

image source

Ethernet silicon types:
1) Merchant Silicons : Some companies develop chips to be used inside Vendor's products, these are highly specialized Ethernet focused chips that stands in-between general commercial silicons (like general purpose CPUs) and vendor propriety ones , example companies are like Arista, Broadcom ad Mellanox .
2) Vendor chips : some huge vendors develop their own chips for their own specialized devices like routers and switches , most famous are Cisco and Juniper .

CPUs vs FPGAs vs ASICS

The difference between the 3 types are features against tradeoffs , CPUs (like x86) provides the most flexibility and programability vs ASICS providing much faster processing speeds and greater bandwidth with the downside of being not programable while that's true, some of the new ASICs provides and SDK but its very hard to develop for , and mostly these SDKs are used by the Networking vendors instead of the end consumer .

FPGAs are the man in the middle , much faster than CPUs but with more programability than ASICs .

Buffering

There is 2 types of buffers , Shallow on-chip buffer , which is very small , very expensive and then yet very fast , and then there is the second type , the Deep buffer , which can be in the hundreds of Gigabytes , but it would be off-chip and would be much slower , you have both in a Chassis router but each handling a particular scenario .

Following is Pete's (Pete Lumbis) Opinion on which buffer should handle which type of traffic .
A) Deep buffer : should handle long stance transmission , because microsecond congestion on such port would have big impact on RTT of TCP, mostly are measured in Giga Bytes, and mostly off-chip .
B) Shallow buffers : everything else , mostly measured in Mega bytes, mostly on-chip.

Shallow Buffer on Trident3 - Broadcom

One of the hardest topics to understand is buffering ,

When is buffer used :

1- When Egree port traffic is larger than Ingress ports - 2 Inputs , 1 output ports , if buffer is full , this would provide Overrun packets .
2- When speed changes , input 100G to output 10G port for example .
3- When you need to Store and Forward .
4- If there is a Burst of traffic that is bigger than th Pipeline speed of processing , depends on the packet header size .

When buffer is not used :

1- Cut through traffic , so traffic coming in and directly being forwarded out with input and output at same speed.

Form Factors

Form factor needs differs from a network addressed to a Cloud provide , hyper scaler , enterprise, or a backbone service provider .

Most Cloud providers and Hyper scalers tends to use a 1RU form factor switches , on the other hand enterprise core switches and service provider's backbone routers tend to be a large multi-line card chassis, in here we are focusing more on the bigger form factor as it provides many concepts to get familiar with .

Lets take a Juniper MX2020 service router , the same functionality are implemented in many vendors and models but the naming may differ a little bit , but they all tend to provide the same functionality :

Routing Engine

Juniper calls it Routing Engine cards , Cisco calls it Route Processor , these are responsible for handling the Control plane in the legacy way including routing protocols , device management, logging , monitoring, and any functionality besides pure packet forwarding , they establish the peerings, hold the routing tables, apply the route selection derive the RIB and then they translate if into the FIB table that gets cascaded to the Forwarding engines .

They usually have some redundancy features such as ISSU(), Non stop forwarding and so on .

The separation between the Routing engines and the Forwarding plane is useful in case of failure of both redundant Routing Engines , you would lose access to the device and the peering would stop but traffic would still transit through the chassis normally without drops as the Forwarding engines (Ports Cards) would have a copy of the last FIB, the only issue would be that FIB would be outdated .

Forwarding Engine

Refers to the processors on the Line Cards (Ports cards) , a line card may include one or more of these forwarding processors (ASICs), Juniper calls them PFE(Packet Forwarding Engine) , Cisco has multiple names for these such as NPU(Network Processing Unit) , or PSE(Packet Switching Engine) .

These are responsible for packet header manipulation, classification, scheduling, policing, rewrite, replication, filtering, accounting, fine-grained traffic steer‐ ing, mirroring, sampling, unicast and multicast RPF checks, class-based forwarding, and many others.

Fabric Processors

When you have a chassis with multiple line cards, you will mostly find these switch fabrics mediating traffic traversing in-between the Line cards (Forwarding Engines).

Backplane

The chassis connectivity lines between all three Routing Engines, Line Cards and Switch fabrics .

Further

These cards would show connected into the backplane serial but actually the connectivity between these cards are more like a Spine/Leaf topology of a datacenter just compacted into a Chassis form , looks like this logically

ASICS provider

Broadcom

Broadcom has a wide range of ASICs for Networks , we are going to focus on the more relevant ones to our article , there are 2 families :
1- StrataSGX : Mostly Data Center and 1RU form factor , and you have 2 lines :

Feature rich (More Enterprise) : Trident and Maverick lines of chips .
High speed (Service Provider) : Tomahawk line of chips . 2- StrataDNX : Deep buffer Chassis chips , they mostly live on line cards :
Buffers and Medium speed : Aurad, Qumran, Jericho

Cisco

Cisco has lately unveiled their Silicon One family of chips targeted towards all fields including data centers , hyperscalers , enterprises and service providers , currently you can see the Q200 Silicon one chip used in the Cisco 8000 family of routers.
Cisco sourced image

Mellanox

mellanox is now a part of Nvidea , there chips are more to be found inside White Boxes rather than vendor chassis .
They have only one family for Ethernet chips :

Spectrum : low latency , feature rich

Intel

In 2019, Intel acquired Barefoot , the maker of the Tofino silicon which is directed towards the Datacenter Fabric market , as of 2022 , they have 3 generations of the Tofino chips family , with P4 programable chips, Intel calls its chips IFPs (Intelligent Fabric Processors), to put it into presepctivet , some Arista (7170 Series) ad Cisco (Nexus 3400 family - Top of rack) Products have already implemented these chips, following is the difference between the 3 generations :
Intel Source Page

References :

Huawei Cloud Engine Ansible

Amr ElHusseiny — Thu, 25 Aug 2022 09:35:00 +0000

Intro

I have been working with the Huawei Cloud Engine series of switches for the Data Center, and one of the great things I was introduced to is the efficient usage of Ansible+Netconf to configure Legacy network devices instead of the trial and error handling way of Python+SSH , which you had to anticipate the delay of the SSH session and to handle a console output using Regex which while very customizable , when it comes to production environments , your supervisors would much prefer a tried and tested technology backed up by the vendor itself .

what we will be discussing

Here I will share the steps I was able to start my journey in configuring the Cloud Engine switches using Ansible .

Steps

1st I would advise you to configure an EVE-NG Community Environment or a GNS3 environment so you are able to test your playbooks (Ansible) before deploying them to the production environment .

1- EVE-NG Installation :

For my deployment , I will be using EVE NG , you can find the download page (https://www.eve-ng.net/index.php/download/) , download the Community version , as it won't require a retail license .

You can download it as ISO to install or and OVF to run directly , for me , I am running it on ESXI machine .

After you downloaded and installed EVE-NG , you should be able to reach the eve machine using SSH 1st to configure it , default SSH user is "root" and default password is "eve" .

When you login to SSH , you will follow a series of configuration prompts for DNS , NTP and so on , after which you will be able to login to the EVE-NG Web portal to start using the Emulation software .

On the Web portal , the default user is "admin" and password is "eve" .

2- Huawei Cloud Engine EVE Image :

In order to work with EVE , you will have to use a preconfigured image , for our device , please follow the following link to download the Huawei CE12800 VM that we will use :
https://forum.huawei.com/enterprise/en/run-ce12800-ne40e-in-eve-ng/thread/653457-861

3- Our lab setup :

After you followed the above guide to add the Huawei Cloud Engine image to EVE , we will be able to create my lab setup :

A) Ansible Machine

I created an ESXi CentOS machine to run the Ansible playbook from , you can install locally on your laptop or machine if you would like , we will be able to reach our Huawei switch from our local Home network .

B) On EVE Portal

Add new Lab
Go to left side node , "Add an object -> Node -> search Huawei , you find Cloud Engine 12800" , click on it , you will need 2 GB of RAM/Memory reserved per switch to run it :
Next , to bridge the switch to our local Network , so we can access it from our local Laptop or in my case from my CentOS VM , you need to go to "Add an object -> Network -> Management (Cloud0) " , and connect one of the switches interface to the cloud (ex: GE1/0/0) , and right click on the switch then choose start :

C) Configuring Huawei Cloud Engine for Netconf :

Hover over the device on EVE to see the IP and Port number you can console to the device using :

'$ telnet 192.168.1.109 32769'

On device , paste the following configuration to add a user and start the Netconf Service :



# Switch Configuration
system
!
interface GE 1/0/0
undo portswitch
undo shutdown 
ip address 192.168.1.130 24
quit
!
ip route-static 0.0.0.0 0 192.168.1.1 
!
netconf
protocol inbound ssh port 830
!
ssh user client001
!
aaa
local-user client001 password irreversible-cipher SetUesrPasswd@123
local-user client001 service-type ssh
quit
commit
!
ssh server cipher aes128_ctr aes256_ctr aes192_ctr aes128_gcm aes256_gcm
ssh user client001 authentication-type password
ssh user client001 service-type snetconf
!
snetconf server enable
!
commit
!
save

To confirm that you are able to reach network connectivity , ping your gateway and your Ansible hosting device :



<CloudEngine>ping -c 5 192.168.1.1 
  PING 192.168.1.1: 56  data bytes, press CTRL_C to break
    Reply from 192.168.1.1: bytes=56 Sequence=1 ttl=64 time=5 ms
    Reply from 192.168.1.1: bytes=56 Sequence=2 ttl=64 time=3 ms
    Reply from 192.168.1.1: bytes=56 Sequence=3 ttl=64 time=4 ms
    Reply from 192.168.1.1: bytes=56 Sequence=4 ttl=64 time=2 ms
    Reply from 192.168.1.1: bytes=56 Sequence=5 ttl=64 time=3 ms

  --- 192.168.1.1 ping statistics ---
    5 packet(s) transmitted
    5 packet(s) received
    0.00% packet loss
    round-trip min/avg/max = 2/3/5 ms

<CloudEngine>ping -c 5 192.168.1.102 
  PING 192.168.1.102: 56  data bytes, press CTRL_C to break
    Reply from 192.168.1.102: bytes=56 Sequence=1 ttl=64 time=2 ms
    Reply from 192.168.1.102: bytes=56 Sequence=2 ttl=64 time=3 ms
    Reply from 192.168.1.102: bytes=56 Sequence=3 ttl=64 time=3 ms
    Reply from 192.168.1.102: bytes=56 Sequence=4 ttl=64 time=4 ms
    Reply from 192.168.1.102: bytes=56 Sequence=5 ttl=64 time=1 ms

  --- 192.168.1.102 ping statistics ---
    5 packet(s) transmitted
    5 packet(s) received
    0.00% packet loss
    round-trip min/avg/max = 1/2/4 ms

To test the Netconf before moving forward to ansible , from your laptop or Ansible hosting VM , run the following command , you should get a long XML output , this means that we are ready for ansible : $ ssh client001@192.168.1.130 -p 830 netconf

4- Setup of the Ansible machine:

Following are the steps for a CentOS machine

A)Create a virtual Python environment:

Recommended in the time of writing the article to use Python 3.8 and above.

Create a virtual environment , in my case using Python 3.10 :
$ sudo /usr/local/bin/python3.10 -m venv huawei_venv

Activate the created virtual environment , you will find the default Python version is the one you created it with .

The Ansible module we will be using is part of the net_commons collection :



(huawei_venv)$ python --version
Python 3.10.5

# Install Ansible and needed libraries for the lab
(huawei_venv)$ sudo /usr/local/bin/python3.10  -m pip install ansible ansible-core ncclient jxmlease xmltodict ansible-pylibssh

# Check Ansible version 
(huawei_venv)$ ansible --version
ansible [core 2.13.3]
  config file = None
  configured module search path = ['/home/amroashram/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.10/site-packages/ansible
  ansible collection location = /home/amroashram/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.10.5 (main, Aug  8 2022, 17:01:37) [GCC 8.5.0 20210514 (Red Hat 8.5.0-15)]
  jinja version = 3.1.2
  libyaml = True

# Double confirm that the Netcommons collection is installed 
(huawei_venv)$ ansible-galaxy collection install ansible.netcommon

B) lets create a simple Inventory and Playbook to run on our current switch :

Inverntory.yaml



# inventory.yaml
all:
  # ----- Variables for the whole inventory 
  vars:
    ansible_network_os: community.network.ce
    ansible_user: client001
    ansible_password: "SetUesrPasswd@123"
    ansible_ssh_common_args: '-o ProxyCommand="ssh -W %h:%p -q bastion01"'
    # log_path: ansible.log 

  # ----- Group names and relations to hosts 
  children:
    network:
      children:
        huawei:
          hosts:
            leaf_1:
              ansible_host: 192.168.1.130

Playbook.yaml : will be adding a vlan (20) to the switch



# Playbook.yaml
- name: test
  hosts: leaf_1
  connection: ansible.netcommon.netconf
  gather_facts: no

  tasks:
  - name: add_VLAN
    community.network.ce_vlan:
      vlan_id: 120
      name: WEB_Vlan
      description: "hello"

To run the playbook on the inventory :
$ ansible-playbook -i huawei_cloud_engine_inventory.yaml huawei_cloud_engine_playbook.yaml

Output confirming change has been done as follows :



 bash

PLAY [test] ***********************************************************************************************************************************************

skipping: no hosts matched

PLAY RECAP ************************************************************************************************************************************************

(huawei_venv) [amroashram@centos_proxy huawei_cloud_engine]$ ansible-playbook -i huawei_cloud_engine_inventory.yaml huawei_cloud_engine_playbook.yaml 

PLAY [test] ***********************************************************************************************************************************************

TASK [add_VLAN] *******************************************************************************************************************************************

changed: [leaf_1]

PLAY RECAP ************************************************************************************************************************************************

leaf_1                     : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0 

  
  
  On switch :


<CloudEngine>display vlan 

  
  
  VID          Ports                                                          


1         UT:GE1/0/1(D)      GE1/0/2(D)      GE1/0/3(D)      GE1/0/4(D)


                GE1/0/5(D)      GE1/0/6(D)      GE1/0/7(D)      GE1/0/8(D)


                GE1/0/9(D)


 120     

  
  
  VID  Type     Status  Property  MAC-LRN STAT    BC  MC  UC  Description


1 common   enable  default   enable  disable FWD FWD FWD VLAN 0001


 120 common   enable  default   enable  disable FWD FWD FWD hello

Further Ansible capabilities

To check further usage of the netcommon collection with Cloud Engine , you have the following modules available to do changes on the switches :

Handy command , if you do not know what a module does or how to use it in the Playbook , you can always use : $ ansible-doc community.network.ce_vlan
Modules : refer back to link (https://docs.ansible.com/ansible/latest/network/user_guide/platform_ce.html) for up to date modules . community.network.ce_aaa_server community.network.ce_aaa_server_host community.network.ce_acl community.network.ce_acl_advance community.network.ce_bfd_global community.network.ce_bfd_session community.network.ce_bfd_view community.network.ce_bgp community.network.ce_bgp_af community.network.ce_bgp_neighbor community.network.ce_bgp_neighbor_af community.network.ce_dldp community.network.ce_dldp_interface community.network.ce_eth_trunk community.network.ce_evpn_bd_vni community.network.ce_file_copy community.network.ce_info_center_debug community.network.ce_info_center_global community.network.ce_info_center_log community.network.ce_info_center_trap community.network.ce_interface community.network.ce_interface_ospf community.network.ce_ip_interface community.network.ce_lacp community.network.ce_link_status community.network.ce_lldp community.network.ce_lldp_interface community.network.ce_mlag_config community.network.ce_netconf community.network.ce_ntp community.network.ce_ospf community.network.ce_ospf_vrf community.network.ce_reboot community.network.ce_sflow community.network.ce_snmp_community community.network.ce_snmp_target_host community.network.ce_snmp_user community.network.ce_static_route community.network.ce_static_route_bfd community.network.ce_switchport community.network.ce_vlan community.network.ce_vrf community.network.ce_vrf_af community.network.ce_vrf_interface community.network.ce_vrrp community.network.ce_vxlan_tunnel community.network.ce_vxlan_vap

References :

EVE_NG Installation : https://www.eve-ng.net/index.php/documentation/installation/virtual-machine-install/
Huawei Cloud Engine EVE Image : https://forum.huawei.com/enterprise/en/run-ce12800-ne40e-in-eve-ng/thread/653457-861
Huawei Cloud Engine Netconf Configuration : https://support.huawei.com/enterprise/en/doc/EDOC1100198823/4b902677/example-for-establishing-communication-between-the-nms-and-a-device-using-netconf
Cloud Engine Netcommons Collection Doc : https://docs.ansible.com/ansible/latest/network/user_guide/platform_ce.html
Python 3.10 veneer : https://docs.python.org/3/library/venv.html

Python's Netmiko template

Amr ElHusseiny — Sun, 17 Jul 2022 09:14:05 +0000

The following is a ready to go netlike template , you just need to choose your way of authentication and the device type , you can find the supported devices on both the documentation and the Git for the Netmiko library :

Netmiko Source Code : Netmiko Github
API reference (must check for more functionality) : Netmiko Doc

# netmiko_starter.py 
from netmiko import ConnectHandler

device_dictionary = {
                    'device_type': 'DEVICE_TYPE',  
                    'host': 'DEVICE_IP_ADDRESS',
                    'username': 'USERNAME',
                    'password' : 'PASSWORD',
                    # 'verbose': False ,
                    # 'session_log': 'log.txt', 
                    # 'global_delay_factor': 2, 
                }
net_connect = ConnectHandler(**device_dictionary)
output = net_connect.send_command("show version")
net_connect.disconnect()

Comments on the code :

uncomment verbose if you want to disable Netmiko's Logs .
uncomment session_log to debug the connection failures .
uncomment _ global_delay_factor_ to add delay when waiting for commands output , very useful if the devices are slow , note that this is not in seconds but in multiple , so "2" mean multiply the default delay by 2 and so on

For more functionality , like the types of devices supported , please check the API docs .

Also check the textfsm templates part of the guide , this helps a lot with parsing the output instead of using the normal Regex way , it supports a lot of well known devices outputs .

good luck

DEV Community: Amr ElHusseiny

Linux Networking Part 1 : Kernel Net Stack

Introduction to series

This article in a pinch

Part 1 : Linux Network Stack

1) Ring Buffers

2) Socket Buffers (sk_buff)

3) Kernel Interrupts (IRQ vs SoftIRQ)

3) Other quick concepts

Network flow in brief

Keywords

Commands Summary

References

gNMI Network Automation (Part 3) : gNMI Telemetry, Telegraf & InfluxDB

gNMI Network Automation (Part 3) : gNMI Telemetry, Telegraf & InfluxDB

Introduction

Brief about topology

Running the topology

Telegraf (Collector)

InfluxDB (Time Streaming Database)

Running the Telegraf and InfluxDB contianers

Graphs on InfluxDB

References

gNMI Network Automation (Part 2) : gNMI Configuration Cisco's IOS-XR

Introduction

OpenConfig vs Vendor

Solving the lack of documentation examples

1st Approach : from device CLI

2nd Approach : Reading Yang files

Deviations

Example Lab

Remote Wire Shark

Useful commands

Conatiners used

References

Network Automation : Simulating network topologies with Containers and gNMI

Introduction

ContainerLab supported platforms (Kinds)

before we start, You can follow the containerlab installation guide here.

Packaging our QCOW into containers

ContainerLab topology template

More labs

Using gNMI & OpenConfig for Automation

Introduction

Limitations of gNMI and OpenConfig

pyang

gNMIc

Useful commands :

References

RPKI , securing BGP Infrastructure - with Lab

Contents

Introduction

What & Why RPKI (Resource Public Key Infrastructure)?

Signing ROAs

what is an ROA ?

How you deploy ?

RPKI-RTR architecture

PDUs Structure (rpki-rtr-pdu)

Protocol workings

Serial Query/Cache Response

Serial Notify

End of data

Reset Query

Applying RPKI

1) Setup a Validator

steps

Allow List

2) Discarding BGP invalids / configuring routers

3) Validating RPKI operations

MANRS

Compulsory Actions to join

References

Extending RFC1918 with RFC-6598 (100.64.0.0/10)

introduction

CGNAT (Carrier Grade NAT)

Special Use IP Addresses

Q&A

References

Automating DrawIO network topology using Python

Intro