Docker and the DF bit

‘Open source tool for determining MTU values in Docker containers’ covers the subject of asymmetric maximum transmission unit (MTU) and provides a basic overview of how Remote.It and Remote.It's open source tools may be used to discover problems with MTU.

We found that Docker containers on macOS ignore the Don’t Fragment (DF) bit. Without support for the DF bit, it is challenging to discover the MTU and MRU of a network connection.

Why is this important? Without knowing the MRU and MTU for a connection, it is hard to create a network connection with maximum efficiency, and sometimes, it may even be hard to create a network connection that works at all.

What is the DF bit?

The DF bit is a single bit field within the Internet Protocol (IP) header of a packet that determines whether a router is allowed to fragment a packet. IP fragmentation is a process that breaks a packet into pieces or fragments so that the resulting pieces can pass through a link in a network connection that has a smaller MTU than can be handled by the original packet size. The fragments are then reassembled by the receiving host. RFC 1191 [Mogul and Deering 1990] specifies the path MTU discovery mechanism (PMTUD), a way to determine the path MTU (PMTU) at any time using the DF bit.

Following RFC1191, PMTUD uses the DF bit to discover the PMTU of a path. A source host initially assumes that the PMTU of a path is the known MTU of the first hop, and sends all datagrams on that path with the DF bit set. If a datagram is too large to be forwarded without fragmentation by a router along that path, then the router will discard the datagram and return an Internet Control Message Protocol (ICMP) Destination Unreachable message with a code that corresponds to "fragmentation needed and DF set". Upon receiving that code and message, which is essentially a "Datagram Too Big" message, the source host reduces the assumed PMTU for the path.

PMTUD will end when the host's estimate of the PMTU is low enough that datagrams can be delivered without fragmentation. The host may also end PMTUD by stopping to set the DF bit in the datagram headers because, for example, it is willing to have datagrams fragmented. Normally, the host continues to set DF for all datagrams, so that if the path changes to a lower PMTU, the new PMTU will be discovered.

Docker and the DF bit

We performed some experiments using a Docker container running on macOS. We summarize these results first, and then show the corresponding macOS Terminal logs.

We used the Remote.It MTU Bouncer tool to send or bounce packets back to us of a certain requested size both with and without the DF bit being set (DF=1). We first ask to be sent a series of packets with DF=1 and length 1770 (which fails, too long), 1470 (succeeds), 1472 (succeeds), 1473 (fails to send, too long). Then we ask for a packet with DF=0 of length 1473 and that packet never arrives, and the same occurs with length 1472. Thus, we suspect that Docker does not maintain the DF bit somewhere between the Docker container and the macOS host OS.

To check this, we then move to the host macOS, eliminating the Docker container, and ask for a packet with DF=1 length 1472 (succeeds), DF=1 length 1473 (fails, too long), then DF=0 length 1473 (which succeeds but previously failed inside the Docker container).

We can conclude that somewhere the DF bit is not being communicated somewhere between the macOS Docker container and host macOS.

Here is the macOS Terminal log corresponding to the experiments and results described above:

~ # nc -u bouncer.remote.it 9999
1770
On size 1770 (1798 MTU dfrag=1) sender failed to send with error Message too long code 90
17
Must be between 20 and 65535 (add f on end to allow fragmentation)
1470
1470 (1498 MTU DF=1)
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Must be between 20 and 65535 (add f on end to allow fragmentation)
1472
1472 (1500 MTU DF=1)
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Must be between 20 and 65535 (add f on end to allow fragmentation)
1473
On size 1473 (1501 MTU dfrag=1) sender failed to send with error Message too long code 90
1473f
1472f
1472 (1500 MTU DF=0)
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C~ #
~ # exit
ops@ops-mac-mini ~ % nc -u bouncer.remote.it 9999
1472
1472 (1500 MTU DF=1)
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Must be between 20 and 65535 (add f on end to allow fragmentation)
1473
On size 1473 (1501 MTU dfrag=1) sender failed to send with error Message too long code 90
1473f
1473 (1501 MTU DF=0)
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
ops@ops-mac-mini ~ %

Note in this case, when we connect to a server located at bouncer.remote.it using netcat and you hit return after entering the netcat command, the terminal console waits for you to enter something. In the above example, we asked the server to send packets of length 1770, 17, 1470, 1472, 1473, 1473f (the f flag sets DF=0), 1472f, 1472, 1473, 1473f, as we probed the path to determine the MTU.

What is Docker?

Docker allows you to package and run an application in a loosely isolated secure environment called a container. The isolation and security allow you to run many containers simultaneously on a single host. Containers are lightweight and possess everything needed to run the application, so you do not need to rely on anything installed on the host. You can share containers with anyone and ensure that the same container that works in the same way everywhere.

Docker is written in the Go and takes advantage of several features of the Linux kernel to perform its functions. For example, Docker uses a technology called namespaces to provide the isolated workspace that forms the container. When you run a container, Docker creates a set of namespaces for each container.

Docker thus has some characteristics that are inherited from Linux and some characteristics that are determined by or inherited from the host OS, in our case macOS. This “split personality”, can cause problems and this article describes one of problems.

Why use Remote.It with Docker?

Remote.It is a powerful network connection tool. Remote.It can connect any two hosts or devices using a peer-peer connection with an agent on each host. Remote.It can also connect using a web-based tool and a proxy to any target host, using a single agent on the target host.

Remote.It can be used to connect containers within a host, containers in different hosts on the same network, containers on different networks, or even containers in different data centers, for example.

As an example of a network connection that would be difficult to create by other methods, suppose container 1 has a default private IP address 172.17.0.1 and container 2 also has the same default IP address 172.17.0.1. Remote.It allows you to connect these two containers.

Docker asymmetric MTU

If we run Remote.It in a Docker container running on macOS we see the following:

Connected to ID [e-mail] - f3:25:9f:01:33:b9:f1:ed at 192.168.2.21:37020
Session Max Packet Detected 1450 (MTU) Max Data Tunnel Size 1386 bytes.
Session Max Packet Received 1500 (MRU) Max Data Tunnel Size 1436 bytes.
last packet from Session peer 0 seconds ago
retry at count 0

Note that we have an asymmetric MTU: the container has an MTU of 1450 bytes (allowing to a tunnel payload of 1386 bytes with a header and encryption overhead of 64 bytes) but an MRU of 1500 bytes (with the same 64-byte overhead).

The asymmetric MTU was discovered using the DF bit. Without support for the DF bit the discovery of MTU and asymmetric MTU, or different MTU and MRU, would be very difficult. As outlined earlier, you also can contend with the support of the DF bit in a macOS Docker container.

Summary

We described the problem we found with the support of the DF bit in a macOS Docker container. We also described MTU, how a path could have asymmetric MTU, and an example of an asymmetric MTU situation in a Docker container, and why it is important to support the DF bit in order to discover MTU and asymmetric MTU/MRU. We described how to discover an asymmetric MTU using Remote.It and the use of MTU Bouncer.