How I Fixed an RFC 2385 TCP-MD5 Protocol Bug in Scapy: An Open-Source Deep Dive

Sakhi Hussain — Fri, 13 Nov 2026 15:20:21 +0000

Scapy issue #5229 came down to one line: tcp_bytes[12] >> 4. That line reads the TCP header length. Give it a UDP layer with no payload, though, and the byte it needs isn't there. The caller gets an IndexError without a hint that they passed the wrong layer.

I fixed that in Scapy PR #5268.

Reproducing the bug

from scapy.layers.inet import IP, UDP, calc_tcp_md5_hash

packet = IP() / UDP()
calc_tcp_md5_hash(packet[UDP], b"example-key")

The packet[UDP] part matters here. It pulls out the UDP layer, which serializes to just eight bytes in this example. The old code reached for byte index 12 to read the high four bits, where TCP stores its header length in 32-bit words. That's where the call failed.

The fix

I added two checks before that byte access. Here's the relevant part of the merged diff.

if not isinstance(tcp, TCP):
    raise TypeError("calc_tcp_md5_hash requires a TCP layer")

tcp_bytes = bytes(tcp)
if len(tcp_bytes) < 20:
    raise ValueError("Truncated TCP header")
doff = tcp_bytes[12] >> 4

The first check catches the wrong layer before serialization. If the caller does pass a TCP layer, the second check makes sure it serializes to at least 20 bytes, the size of a TCP header without options. Run the UDP example with the fix and the error tells you exactly what to pass: TypeError: calc_tcp_md5_hash requires a TCP layer.

What I tested

In test/scapy/layers/inet.uts, I added three calls that should raise TypeError:

IP() / UDP(), which passes the whole IPv4 packet to the helper.
UDP(), which passes a standalone UDP layer.
b"invalid_layer", which passes a Python bytes object.

All three exercise the type check. The test file already had an IPv4/TCP case that compares the generated MD5 option with an expected value. Test diff

Getting it merged

gpotter2 approved and merged the fix into Scapy's master branch on November 4, 2026. Another PR addressed the same issue, and gpotter2 noted there that the project had decided to go with #5268. Merge record · Related PR #5236

DEV Community: Sakhi Hussain

How I Fixed an RFC 2385 TCP-MD5 Protocol Bug in Scapy: An Open-Source Deep Dive

Reproducing the bug

The fix

What I tested

Getting it merged