DEV Community: Mishal Shah

Running DPDK Forwarding Applications with Pktgen-DPDK

Mishal Shah — Thu, 16 Jul 2020 19:09:05 +0000

This article is co-authored by Mehnaz Yunus

As part of the evaluation stage of our bachelor thesis, we set up a testbed for running forwarding applications in DPDK and with Pktgen-DPDK as the traffic generator. In this blog, we aim to cover

how to run Pktgen-DPDK using Lua scripts in some complex scenarios
how to run L2fwd, L3fwd (with exact match and longest prefix match), and L3fwd power DPDK applications.

We weren't able to find a lot of resources for setting up these tests and with this blog, we aim to bridge the gap.

DPDK and Pktgen-DPDK

The Data Plane Development Kit is an Open source software project managed by the Linux Foundation. It provides a set of data plane libraries and network interface controller polling-mode drivers running in userspace. This way, the NIC is directly accessible by the DPDK application. The advantage of using DPDK is that you're able to utilize the link speed completely whereas, in the standard packet processing, the link goes underutilized. Thus, high performance can be achieved with DPDK.

Pktgen-DPDK is a software-based traffic generator powered by DPDK. Traffic generators are often used to simulate various situations and test the performance of the application.

Test Setup

All the tests were done with two systems, one as the System Under Test which is a bare metal server, and the other one as a generator which is a VM. The SUT has two NUMA sockets where each socket has two 14-core Intel Xeon Gold 5120 2.20 GHz processors, which has four Intel I350 10 Gigabit NICs and four Ethernet Controller X710/X557-AT 10GBASE-T NICs. The generator has two 12-core Intel Xeon Gold 5120 2.20 GHz processors, 94GB memory, and one Intel 82540EM Gigabit NIC and Ethernet Controller X710/X557-AT 10GBASE-T NICs. Each system runs Ubuntu 20.04 LTS.

Installation of both DPDK and Pktgen-DPDK was done by compiling from source. igb_uio kernel driver was used and 2048 huge pages were allotted. Our systems had 4 NICs so we could utilize them, if your system doesn't have that many NICs, you could use tapped interfaces. Here's a diagrammatic representation of the setup.

Lua Scripts with Pktgen-DPDK

We used Lua scripts to automate the process of setting parameters and configuring flows in Pktgen-DPDK. Pktgen-DPDK can be run on the generator using the command shown below. It is run in promiscuous mode with the first four cores (0,1,2,3), and 4 memory channels. The Lua script for each application is mentioned using the -f option. The port mapping to cores is done as:

core 1 handles port 0, 1 RX/TX
core 2 handles port 2, 3 RX/TX.

💡 Note: The core 0 is used for keyboard, timer, and screen output and hence shouldn't be used for port mapping. pktgen-dpdk with Lua script should be run from the pktgen directory only, otherwise, it gives an error.

sudo ./app/x86_64-native-linux-gcc/pktgen -c 0xf -n 4 -- -P -m 1.[0-1] -m 2.[2-3] -f /path/to/script.lua

Situation 1: Long-Running Flow for Certain Time

We were required to run tests where the packets were sent for a fixed time period. Since pktgen by default doesn't support timers, we used os functions for the same.

local port = "0";
local send_for_secs = 120;
pktgen.start(port);
local start_time = os.time();

while os.difftime(os.time(), start_time) < send_for_secs do
    sleep(1);
end

pktgen.stop(port);

where send_for_secs is the time for which we want to send.

💡 Note: sleep(send_for_secs) was not used because it would block Pktgen's output refreshing. So continuous check after a second and checking until the difference reaches the send_for_secs is done here.

Situation 2: Different Packet Size and Rate

We also wanted to test with different packet sizes and rates (rate in pktgen-dpdk is to be given as a percentage of the line rate). Since changing the packet size or rate in the Lua script after every run is tedious, we ask for user input for the packet size and rate at runtime using the script below.

local pktSize = tonumber(pktgen.input("Enter the size of packets to send: "));
if pktSize == nil then
    pktSize = 64;
end
pktgen.set(port, "size", pktSize);

local rate = tonumber(pktgen.input("Enter the sending rate: "));
if rate == nil then
    rate = 100;
end
pktgen.set(port, "rate", rate);

Forwarding Applications in DPDK

We mention the steps to run the forwarding application and a corresponding Lua script to set relevant parameters. The command to run Pktgen-DPDK with the Lua script has been given in the previous section.

l2fwd

Layer 2 forwarding (switching) does not need IP addresses to work. Thus, we set only the MAC address in the packets sent from pktgen-DPDK.

Start the l2fwd application on the SUT: The l2fwd example can be run with the following command on the SUT. This command starts l2fwd on four cores and with four ports. The SUT has two NUMA sockets but we use only one. The even-numbered cores are on socket 0 and we use the first four cores on it, i.e, cores 0,2,4,6. Thus the core mask is given as 0x55. The port mask is 0xf as we use four ports.

sudo ./${RTE_SDK}/examples/l2fwd/${RTE_TARGET}/app/l2fwd -c 0x55 -n 4 -- -p 0xf

Start pktgen-dpdk on the generator with Lua script: Get the MAC address of the ports by starting the l2fwd application on the SUT. Set them as the destination MAC address for the ports 0 and 2 in pktgen through the Lua script shown below.

local port0 = "0";
local port2 = "2";

local port0_mac = "3C:FD:FE:7C:BC:E0";
local port2_mac = "3C:FD:FE:7C:BC:E2";

pktgen.set_mac(port0, "dst", port0_mac);
pktgen.set_mac(port2, "dst", port2_mac);

-- code for taking user input for packet size and rate
-- setting the packet size & rate

pktgen.start(port0);
pktgen.start(port2);

-- code to check time to run

pktgen.stop(port0);
pktgen.stop(port2);

l3fwd

A layer 3 forwarder matches the destination IP of an incoming packet to the entries stored in its forwarding table and updates the layer 2 header before sending the packet to the next-hop.

Longest Prefix Match

For the LPM method, an LPM object is used to emulate the forwarding stage for IPv4 packets. The LPM object is used as the routing table to identify the next hop for each input packet at runtime. The example can be run with the following command on the SUT.

sudo ./${RTE_SDK}/examples/l3fwd/${RTE_TARGET}/app/l3fwd -c 0x55 -n 4 -- -P -p 0xf --config=(0,0,0),(1,0,2),(2,0,4),(3,0,6)

The IP addresses obtained from the source code of l3fwd are set as shown below.

local port0 = "0";
local port1 = "1";
local port2 = "2";
local port3 = "3";

pktgen.set_ipaddr(port0, "dst", "198.18.1.1");
pktgen.set_ipaddr(port0, "src", "198.18.0.1");
pktgen.set_ipaddr(port1, "dst", "198.18.0.1");
pktgen.set_ipaddr(port1, "src", "198.18.1.1");

pktgen.set_ipaddr(port2, "dst", "198.18.3.1");
pktgen.set_ipaddr(port2, "src", "198.18.2.1");
pktgen.set_ipaddr(port3, "dst", "198.18.2.1");
pktgen.set_ipaddr(port3, "src", "198.18.3.1");

-- code for taking input of packet size, rate from the user
-- setting the packet size & rate

pktgen.start(port0);
pktgen.start(port2);

-- code to check time to run

pktgen.stop(port0);
pktgen.stop(port2);

Exact Match

In the exact match lookup method, the forwarding is based on a hash for which a hash object is used to emulate the flow classification stage. The example can be run with the following command on the SUT. The -E parameter selects the exact match method.

sudo ./${RTE_SDK}/examples/l3fwd/${RTE_TARGET}/app/l3fwd -c 0x55 -n 4 -- -P -E --parse-ptype -p 0xf --config=(0,0,0),(1,0,2),(2,0,4),(3,0,6)

From the code for the l3fwd application, the exact match works only with TCP packets and thus we set the protocol as TCP in the packets generated from pktgen. The exact match is based on the combination of IP addresses and port numbers, as seen in the routing table in the code. We set the IP addresses and destination and source ports as shown below obtained from the source code.

local port0 = "0";
local port1 = "1";
local port2 = "2";
local port3 = "3";

pktgen.set_ipaddr(port0, "dst", "201.0.0.0");
pktgen.set_ipaddr(port0, "src", "200.20.0.1");
pktgen.set_ipaddr(port1, "dst", "101.0.0.0");
pktgen.set_ipaddr(port1, "src", "100.10.0.1");

pktgen.set_ipaddr(port2, "dst", "211.0.0.0");
pktgen.set_ipaddr(port2, "src", "200.40.0.1");
pktgen.set_ipaddr(port3, "dst", "111.0.0.0");
pktgen.set_ipaddr(port3, "src", "100.30.0.1");

pktgen.set(port0, "sport", 12);
pktgen.set(port0, "dport", 102);
pktgen.set(port1, "sport", 11);
pktgen.set(port1, "dport", 101);

pktgen.set(port2, "sport", 12);
pktgen.set(port2, "dport", 102);
pktgen.set(port3, "sport", 11);
pktgen.set(port3, "dport", 101);

-- code for taking input of packet size, rate from the user
-- setting the packet size & rate

pktgen.set_proto("all", "tcp");

pktgen.start(port0);
pktgen.start(port2);

-- code to check time to run

pktgen.stop(port0);
pktgen.stop(port2);

l3fwd-power

The L3 Forwarding with Power Management application is an example of power-aware packet processing using the DPDK. The example can be run with the following command on the SUT.

sudo ./${RTE_SDK}/examples/l3fwd-power/${RTE_TARGET}/app/l3fwd-power -c 0x55 -n 4 -- -P -p 0xf --config=(0,0,0),(1,0,2),(2,0,4),(3,0,6)

The flow configured on pktgen with IP addresses can be done as below. The IP addresses are obtained from the source code of l3fwd-power. It runs the longest prefix match by default.

local port0 = "0";
local port1 = "1";
local port2 = "2";
local port3 = "3";

pktgen.set_ipaddr(port0, "dst", "1.1.1.1");
pktgen.set_ipaddr(port0, "src", "2.1.1.1");
pktgen.set_ipaddr(port1, "dst", "2.1.1.1");
pktgen.set_ipaddr(port1, "src", "1.1.1.1");

pktgen.set_ipaddr(port2, "dst", "4.1.1.1");
pktgen.set_ipaddr(port2, "src", "3.1.1.1");
pktgen.set_ipaddr(port3, "dst", "3.1.1.1");
pktgen.set_ipaddr(port3, "src", "4.1.1.1");

pktgen.set(port0, "sport", 12);
pktgen.set(port0, "dport", 102);
pktgen.set(port1, "sport", 11);
pktgen.set(port1, "dport", 101);

pktgen.set(port2, "sport", 12);
pktgen.set(port2, "dport", 102);
pktgen.set(port3, "sport", 11);
pktgen.set(port3, "dport", 101);

-- code for taking input of packet size, rate from the user
-- setting the packet size & rate

pktgen.set_proto("all", "tcp");

pktgen.start(port0);
pktgen.start(port2);

-- code to check time to run

pktgen.stop(port0);
pktgen.stop(port2);

Conclusions

In this article, we looked at running various forwarding applications of DPDK with pktgen-dpdk as the traffic generator with four ports. The main challenge we faced while trying to run these applications was a lack of proper documentation in setting up various cases with pktgen and making sure the traffic forwards on the application and reaches back to the other port. If you do not configure the IP/MAC address properly, it is possible that the packets do not reach the other port but come back on the same port. We hope with this blog, you'll be able to configure your flows with ease.

Both DPDK and pktgen-dpdk require a bit of work to set up and understand how the applications work, and undoubtedly there is a good learning curve. The DPDK documentation is quite detailed on running the DPDK applications and thus, we've not gone much into deep covering it but concentrated on pktgen-dpdk.

The complete Lua scripts can be found here

Acknowledgment

We thank our project guide, Dr. Mohit P. Tahiliani, and mentor Leslie Monis for their continuous guidance throughout the course of the project and our project teammate Pavan Vachhani for his valuable contribution to the final project.

References

DPDK Sample Application Guide: l2fwd, l3fwd, l3fwd-power
Pktgen-DPDK
DPDK Source Code: https://github.com/DPDK/dpdk

An introduction to Chaos Engineering

Mishal Shah — Wed, 08 Jul 2020 09:56:04 +0000

Introduction

Many of us would have heard about the two major WhatsApp outages recently when it was down for about an hour each time. For the general public, WhatsApp was down for an hour. But for software engineers, WhatsApp was back within an hour. This made me wonder what it was, that enabled the engineers working at WhatsApp to restore the services so fast.

What if I tell you, it was a very normal turbulent production failure for them. Yes, huge companies like Netflix, Facebook, and Google use a method of engineering called Chaos Engineering.

To explain Chaos Engineering in layman terms: To be good at something, if one practices it in extreme conditions then it would get easier for the person to perform in normal conditions. Similarly, if a software engineer has seen extreme conditions in production, it becomes a challenge for him/her to code in such a way that their service fails if such an extreme condition occurs. If at all it comes, they would have experienced it before and so it becomes easy to bring the services back.

Principle of Choas Engineering

The main idea of Chaos Engineering is to intentionally inject failures so as to be prepared for the worst conditions. A formal definition for Chaos Engineering given by Netflix:

Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.

Chaos Engineering can be thought of as the facilitation of experiments to uncover systemic weaknesses. These experiments follow four steps:

Start by defining ‘steady state’ as some measurable output of a system that indicates normal behavior.
Hypothesize that this steady-state will continue in both the control group and the experimental group.
Introduce variables that reflect real-world events like servers that crash, hard drives that malfunction, network connections that are severed, etc.
Try to disprove the hypothesis by looking for a difference in the steady-state between the control group and the experimental group.

The harder it is to disrupt the steady-state, the more confidence we have in the behavior of the system. If a weakness is uncovered, we now have a target for improvement before that behavior manifests in the system at large. In layman terms: Break your system, see the difference between steady-state and experimental state, lesser the difference, lesser it will break in production. If the difference is more, you have a target for improvement before actually experiencing it in production.

Practical Use

Netflix, a pioneer in the field of Chaos Engineering, uses a tool called Chaos Monkey. It randomly picks a server from production deployment on AWS (Amazon Web Services) and kills it. Executives at Netflix knew that server failures are guaranteed to happen and they wanted servers to fail during working-hours so that it could be fixed it in the presence of their engineers. They knew that they could rely on engineers to build resilient solutions if they were given context to expect servers to fail. If they could align their engineers to build services that survive a server failure, then when it actually happened, it wouldn’t be a big deal.

How can it help you?

Suppose your team has built the next wave of advancement the whole world has been waiting for. You are to launch your service but there are questions rumbling in your mind:

Is your application ready for production?
Will the system survive the failures of other companies you are dependent on?
Will the system survive the failure of your own configurations?

The truth is: You can never be sure. There will always be something that can be wrong or will go wrong. Some things aren’t in your control like denial-of-service attacks or network failures. Sometimes, bad things just happen.

Possibly The only solution for it is to build quality software that is resilient to failures. As the saying goes - "Hope for the best, prepare for the worst" But how can your team be sure that your system is ready for production and is ready to face all difficulties/failures? This is where Chaos Engineering comes into the picture.

Does this seem like testing?

It may indeed sound like it, but it definitely isn’t testing. Chaos Engineering is an experiment on the production environment and there is certainly no way to accurately duplicate the production environment at scale (as in testing). The experiment itself is going to have a systemic effect that could change your results, so the only way to accurately build conflict in the system that you have now is to experiment on it.

Conclusion

The purpose of Chaos Engineering is to experience disastrous conditions. This might sound like a difficult task and it does require a lot of creativity but the extra effort is worth it. You must inject failures in your system such that certain parts of your infrastructure becomes unavailable. Some examples are: terminate cluster machines, kill worker processes, delete database tables, cut off access to internal-external services, etc. Later you can also simulate events capable of disrupting steady-state like high latency due to slow networks.

These experiments are a bit difficult but whatever you decide to do, you’ll be surprised at how much you can learn from chaos. There are a lot of tools worth mentioning :

Chaos Monkey - A resiliency tool that helps applications tolerate random instance failures
The Simian Army - A suite of tools for keeping your cloud operating in top form
kube-monkey - An implementation of Netflix's Chaos Monkey for Kubernetes clusters
Gremlin Inc. - Failure as a Service
Pumba - Chaos testing and network emulation for Docker containers (and clusters)
Chaos Toolkit - A chaos engineering toolkit to help you build confidence in your software system
orchestrator - MySQL replication topology management and HA

I hope you got an idea about Chaos Engineering - a powerful approach to build resilient systems!

Migrating Jekyll Blog to Gatsby

Mishal Shah — Wed, 08 Jul 2020 09:44:19 +0000

I had started building my personal website in 2017 with static HTML, CSS, JS and in 2018 based on the available options, I went ahead with Jekyll. Overall, everything was working fine with Jekyll and was smooth. Although, my Twitter feed was filled with Gatsby, and I had seen some stickers named "My Website is faster than yours" of Gatsby while interacting with them for HackVerse Hackathon.

This really made me look at some of the Gatsby sites and it really felt that the websites were fast, and it had the support of PWA, something which I really wanted on my site. Moreover, since it was based on React, I decided to try out to move my personal website from Jekyll to Gatsby.

The Migration

I won't be listing the steps in detail on how I did the migration because there are plenty of blogs out there and the documentation provided by Gatsby is excellent. I could get anything I wanted on the Gatsby documentation. I started looking into the tutorial of Gatsby to get a hang of it and started my new website with the default starter code.

Here are the major steps:

Started looking into the tutorial,
Took the gatsby default starter code,
Added the initial components ( sidebar , post , default ), and set up - the basic styling as per the old website, - Added the content for the non-blog pages ( about , projects , contact , 404 ), - Setup blog page component view (blogPost) from markdown pages using the onCreateNode , createPages APIs, - Learnt a bit about using GraphQL and GraphiQL UI, - Exported all the blog posts from the previous site, - Index Page Display to list all the blog posts ( blogList ),
Added SEO for all pages,
Added automatic deployment with TravisCI,
Added Google Analytics & PWA support,
Some other fixes for overall look & feel.

Here's the lighthouse report of the new website:

GitHub Actions

When GitHub Actions was released in August 2019, I always wanted to give it a try for some project but didn't get enough time to try it out. While I was using Travis CI with this deployment, I thought of giving GitHub Actions a try. Finally!

Here are the steps I followed:

Generated a New Token from GitHub Settings -> Developer Settings -> Personal access tokens with repo access scope.
I was deploying on my root github.io so the name of my repo is mishal23.github.io, I added the token in the repository Settings -> Secrets with the name GH_TOKEN

Created a file named gh-pages-deploy.yml in the directory .github/workflows/ using the Simple Workflow template available under Actions tab on the repository.
Since, the User pages must be built from the master branch, I set my default branch as source on the repository and left the master branch for deployment and hence triggered the workflow only on source branch push,
I configured it to node.js environment with 12.x version, cached the dependencies so that it doesn't have to be installed every time and faster CI,
Configured to initial git settings of user.name and user.email as it is required, used to GH_TOKEN to give the access and then made the deployment run using the following command:

npm run deploy

That's it! Pretty simple! Really like the simplicity of GitHub Actions. Find the complete configuration file here

Conclusion

The migration was definitely worth it, the site is actually blazing fast 🚀 and moreover, got to pick up some new skills. Would recommend anyone to migrate to gatsby. There are certain features and fixes which I plan to do in due time. If you need any further help, feel free to take a look at the repo or contact me If there are any issues you come across, please do open up here or would love it if you fix it :)

Virtual Clinic - An Integrated Care System

Mishal Shah — Fri, 22 May 2020 07:47:18 +0000

My Project

Virtual Clinic was an attempt to bring all the stakeholders of the healthcare system on one platform and make it easier for the patients to communicate and thus improve the operations. It has over 20+ features and a neat user interface.

Demo

The application was deployed on Heroku with CI/CD using Travis CI, please find the link here: http://virtual-clinic.herokuapp.com/

Link to Code

The entire code is open-sourced on my GitHub handle: https://github.com/mishal23/virtual-clinic

How I built it

The entire application is written in Python Framework - Django, with certain packages, unit tests, Travis CI for CI/CD, code coverage, documentation, followed all the Software Engineering Principles. There were many issues that I had faced building this:

multiple role-based access system
ensuring the DRY principle
session keys to maintain
coupling the entire application together with a flow
building a decent front-end
migrating database from sqlite3 to MySQL

Additional Thoughts

The application is no unique, there would be many such existing apps, but this was an application that I built on my own alone completely from scratch, and at a point, only contributing to other projects in a team felt as if I haven't done anything on own and hence I don't know a lot of things. Hence, this was a very special project for me, it gave me the required confidence in my skills and the software engineering principles, the amount of confidence I had after this project to work on any other project was just boosted up. Having made a decent front-end design, automating the deploy, a bit complex backend, documenting, following SE principles - it helped me in all prospects.

If you like my project, please do consider starring the repository or contributing to it either by code, opening issues, or adding more documentation.