Bouncer - Diagnose Network Connection Problems

In this article we describe Bouncer, an open source tool that we created to help solve some of the problems described in the previous two articles. We also describe some of the other networking issues we encountered in working with Docker containers, in particular on macOS. Bouncer is not limited to connections involving Docker containers though.

‘Open source tool for determining MTU values in Docker containers’ covers the subject of asymmetric maximum transmission unit (MTU) and provides a basic overview of how Remote.It and Remote.It tools may be used to discover problems with MTU.

We also found that Docker containers on macOS ignore the Don’t Fragment (DF) bit. Without support for the DF bit it turns out that it is difficult to discover the MTU and MRU of a network connection. Read more about 'Docker and the DF Bit'.

What is Bouncer?

Remote.It Bouncer is a software tool written by Remote.It to help diagnose connection problems. For example, Bouncer can help diagnose problems with MTU and the DF bit. Bouncer is a server that can test the MTU of a connection made using UDP. You connect to the Bouncer server with netcat, or a similar networking tool, and request the return of a UDP packet with a data size that you specify. Bouncer returns a packet with characteristics and properties that specify. For example, you can set the packet to be returned with the DF flag. You can also specify clearing the DF flag if supported in your OS. In our experience, macOS does not currently appear to support setting the DF flag.

You can find Remote.It's open source Bouncer at Github https://github.com/remoteit/MTU_Bouncer provided under the MIT License.

We have also used Bouncer to discover endpoints that do not allow fragmented packets. For example, you can request Bouncer to return a large MTU packet and set the DF bit on to test fragmented packets. If you do not get a response on an otherwise known good connection, then something, somewhere in your path may be dropping fragmented packets.

Bouncer Usage

usage: ./mtu_bouncer [-h] [-v(erbose)] [-d][pid file] [-l listen_tcp_port]
        -h this output.
        -v console debug output.
        -d runs the program as a daemon with optional pid file.
         -l Listen port (defaults to 9999)

Examples of Bouncer use

Run the Bouncer server:

./mtu_bouncer

Connect to the Bouncer server using netcat:

nc -u <serverIP> 9999

Bouncer uses UDP, so you will have to send response, even a null character, to allow Bouncer know your IP address. So, when you hit return after entering the netcat command, the terminal console will wait for you to enter the response. For example, you could use this command to send a newline and cause Bouncer to prompt you for action:

echo -n "/n" | nc -v -u bouncer.remote.it 9999

You can ask the Bouncer server to send you a UDP packet with a payload of specified size. For example, you can successively enter 10, 100, 1472, 1473f to request Bouncer to send packets of size 10 (with DF=1), 100 (DF=1), 1472 (DF=1) and 1473 (with DF=0):

ops@ops-mac-mini ~ % nc -u <serverIP> 9999
10
Must be between 20 and 4000 (add f on end to allow fragmentation)
100
100 (128 MTU DF=1)
..............................................................................
1472
1472 (1500 MTU DF=1)
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
1473
On size 1473 (1501 MTU dfrag=1) sender failed to send with error Message too long code 90
1473f
1473 (1501 MTU DF=0)
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Building Bouncer

Using the makefile in the src directory of the repo should build Bouncer on most Linux or Unix platforms. There is also a win32 project for building on Windows console app.

What is MTU?

The OSI model consists of seven abstract layers: Physical, Data link, Network, Transport, Session, Presentation, and Application. The Physical layer may be Ethernet, for example. Ethernet carries frames that carry the packets.

There is a limit on the size of the Ethernet frame that typically limits the size of data to 1500 bytes. This limit of the link layer is called the MTU, maximum transmission unit.

If the IP layer has a datagram to send, and the datagram is larger than the link layer's MTU, IP performs fragmentation, breaking the datagram up into smaller pieces called fragments so that each fragment is smaller than the MTU.

When two hosts on one network communicate with each other, there is a single network MTU. When two hosts communicate using multiple networks, each link can have a different MTU. The important numbers are not the MTUs of the two networks to which the two hosts are connected, but the smallest MTU of any data link that packets traverse between the two hosts. The smallest network MTU is called the path MTU.

The path MTU between any two hosts is not necessarily constant and may depend on the route being used at any time. Routing need not be symmetric so that the route from A to B may not be the route from B to A, and thus the path MTU need not be the same in both directions. In that case the MRU or maximum receive unit may not be the same as the MTU.

What is the DF bit?

The DF bit is a single-bit field within the Internet Protocol (IP) header of a packet that determines whether a router is allowed to fragment a packet. IP fragmentation is an process that breaks a packet into pieces or fragments, so that the resulting pieces can pass through a link in a network connection that has a smaller MTU than can handle the original packet size. The fragments are then reassembled by the receiving host. RFC 1191 specifies the path MTU discovery mechanism (PMTUD), a way to determine the path MTU (PMTU) at any time using the DF bit.

Following RFC1191, PMTUD uses the DF bit to discover the PMTU of a path. A source host initially assumes that the PMTU of a path is the known MTU of the first hop, and sends all datagrams on that path with the DF bit set. If a datagram is too large to be forwarded without fragmentation by a router along that path, then the router will discard the datagram and return a Internet Control Message Protocol (ICMP) Destination Unreachable message with a code that corresponds to "fragmentation needed and DF set". Upon receiving that code and message, which is essentially a "Datagram Too Big" message, the source host reduces the assumed PMTU for the path.

PMTUD will end when the host's estimate of the PMTU is low enough that datagrams can be delivered without fragmentation. Or the host may end PMTUD by stopping to set the DF bit in the datagram headers because, for example, it is willing to have datagrams fragmented. Normally, the host continues to set DF for all datagrams, so that if the path changes to a lower PMTU, the new PMTU will be discovered.

macOS tool issues

One of the limitations that we found with netcat on macOS was a limit of 1024 bytes in terminal input. This system limit is set as follows, presumably inherited from BSD:

define MAX_INPUT             1024   /* max bytes in terminal input */

This 1024-byte limit makes it harder to use netcat to generate packets large enough to test the MTU of a path. A Github project documents some of other undocumented semantics of Apple's Network framework.

Why use Remote.It with Docker?

Remote.It is a powerful connection tool that can connect any two hosts or devices using a peer-peer connection with an agent on each host. Remote.It can also connect using a web-based tool and a proxy to any target host, using a single agent on the target host.

Developers can be used to connect containers within a host, containers in different hosts on the same network, containers on different networks, or even containers in different data centers for example.

Suppose container 1 has a default IP address 172.17.0.1 and container 2 also has a default IP address 172.17.0.1. Remote.It allows you to connect these two containers.

Summary

We found a problem with support of the DF bit in a macOS Docker container. We also described MTU, how a path can have asymmetric MTU and an example of an asymmetric MTU situation in a Docker container and why it is important to support the DF bit in order to discover MTU and asymmetric MTU/MRU. We described how to discover an asymmetric MTU using Remote.It and the use of MTU Bouncer.