Mustafa ERBAY

Posted on Sep 14 Originally published at mustafaerbay.com.tr

tls on 106 Sockets, tls_stat Zero: kTLS Never Ran

#tls #ktls #linux #nginx

Earlier today, while dumping my server's socket table for the keepalive piece, a word at the end of the lines caught my eye: tcp-ulp-tls. It was on every one of nginx's HTTPS connections. I was on a different subject, so I wrote it down and moved on. This afternoon I opened the note and learned that this stamp, which at first glance says "so nginx is using kernel TLS", is actually the half-finished trace of a feature nobody ever turned on.

Here are the numbers. Ubuntu 24.04, kernel 6.8, nginx 1.30.0, OpenSSL 3.0.13:

$ sudo ss -tnipH > snap.txt
$ grep -c "\"nginx\"" snap.txt
120
$ grep -c tcp-ulp-tls snap.txt
106
$ lsmod | grep ^tls
tls                   155648  106
$ grep -o "rxconf: [a-z]* txconf: [a-z]*" snap.txt | sort | uniq -c
    106 rxconf: none txconf: none
$ cat /proc/net/tls_stat
TlsCurrTxSw                         0
TlsCurrRxSw                         0
TlsTxSw                             0
TlsRxSw                             0
...

Of nginx's 120 TCP sockets, 106 have the TLS module attached: 71 are connections accepted on 443, 35 are connections nginx opened as a client to its own backend (127.0.0.1:8443, a Go service). The tls module's reference count is exactly 106 as well. But every one of them reads rxconf: none txconf: none, and the kernel's own counter, /proc/net/tls_stat, has not seen a single kTLS session since the boot on September 10. Module loaded, attached to every socket, encrypting on none of them.

This post chases that contradiction: where the stamp comes from, why the kernel has an "attached but not working" state at all, what ss and strace show when you really turn it on, and one thing that can happen to you if you turn it on under Ubuntu 24.04. All of it from my own server and from source code; I did not measure speed, and in that section I will quote someone else's number as someone else's number.

The kernel's two-step design

Thinking of Linux kTLS as a single "on" switch is misleading. The kernel documentation describes two separate steps. First, on an established TCP socket, the TLS upper layer protocol is attached with setsockopt(SOL_TCP, TCP_ULP, "tls"). In the documentation's own words, this step merely "allows us to set/get TLS socket options". No encryption starts yet; the handshake completes in userspace, keys come out, and only then does the second step follow: setsockopt(SOL_TLS, TLS_TX, crypto_info) and, if wanted, TLS_RX separately. The transmit and receive paths move into the kernel independently.

The rxconf/txconf fields that ss prints are the state of the second step. In iproute2's ss.c, the tcp_tls_conf() function translates the value from the kernel into four known words: none (TLS_CONF_BASE), sw, hw, hw-record. So the correct reading of a "tcp-ulp-tls rxconf: none txconf: none" line is this: first step taken, second not. Stamp present, key absent.

What happens at the first step on the kernel side is told by tls_init() in net/tls/tls_main.c. If the socket is not ESTABLISHED it returns -ENOTCONN (you cannot attach to a listening socket; the comment says accept would share the ULP context rather than clone it). Then a tls_context is allocated, tx_conf and rx_conf are set to TLS_BASE, and the socket's sk_prot table is swapped. The TLS_BASE table is a copy of TCP's; only three entries differ: setsockopt, getsockopt and close. The functions that send and receive data stay TCP's. That is why all 106 sockets still run at plain TCP speed on the plain TCP path; the encryption is in userspace, in OpenSSL.

Who applies the stamp

There is not a single line mentioning ktls in the nginx configuration; I grepped the nginx -T output. So who takes the first step? I put a plain openssl s_client on the server under strace:

$ strace -f -e trace=setsockopt openssl s_client -connect 127.0.0.1:443 -servername mustafaerbay.com.tr </dev/null
setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(3, SOL_TCP, TCP_ULP, [7564404], 4) = 0
setsockopt(3, SOL_TCP, TCP_ULP, [7564404], 4) = -1 EEXIST (File exists)

7564404 is the byte value of the string "tls" (0x736c74). It is attempted twice, the second bounced by the -EEXIST in the kernel's __tcp_set_ulp(); there is not one call involving SOL_TLS. So the stamp is applied not by nginx but by the libssl underneath it; nginx and s_client use the same library, which is why I see it on both.

In OpenSSL 3.0.13's source this lives in five places: SSL_set_fd(), SSL_set_wfd(), BIO_new_socket(), BIO_connect() and the branch of the connect BIO where the connection is established. The same comment is repeated in each: "ktls_enable doesn't change any functionality of the socket, except changing the setsockopt to enable the processing of ktls_start. Thus, it is not a problem to call it for non-TLS sockets." If the library was built with ktls support (enable-ktls), this call goes to every socket, whether the application wants kTLS or not. Debian turned that option on in the alpha13 package of 3.0.0, in April 2021; one line in the changelog: "Enable ktls." Ubuntu inherits Debian's packaging; under debian/patches in the 24.04 source package, that 2021 patch adding enable => ["afalgeng", "ktls"] to the Linux target definition is still there.

Upstream started discussing these lines in 2022 (issue #19676: "setsockopt TCP_ULP response not checked"); a PR opened on June 26, 2025 and merged into all active branches on October 14, 2025 (#27908, "move ktls_enable() within ktls_start()") moved the ULP attachment to right in front of the second step. I counted the release dates from GitHub tags: 3.0.18, 3.5.4 and 3.6.0 (September-October 2025) still have the old behavior; 3.0.19, 3.5.5 and 3.6.1 (all three on January 27, 2026) do not; 4.0.0 does not either. The distribution side is more interesting: Debian bookworm carries 3.0.20 and trixie carries 3.5.7, both clean. Ubuntu 24.04 stays on 3.0.13 and backports security fixes; this behavioral change was not backported. I verified with three containers on the same server; with --network host they share the kernel and connect to the same nginx:

ubuntu:24.04          OpenSSL 3.0.13   TCP_ULP calls: 2
debian:bookworm-slim  OpenSSL 3.0.20   TCP_ULP calls: 0
debian:trixie-slim    OpenSSL 3.5.7    TCP_ULP calls: 0

The library version decides what gets written to the kernel; the distribution version decides which library you keep. Every TLS socket that stays on Ubuntu 24.04 will carry this stamp, probably until 2029.

I asked the kernel about "doesn't change any functionality"

OpenSSL's comment is correct from the library's point of view: the data path does not change. From the kernel's point of view three things change, and I found all three in the source.

First, __tcp_set_ulp() in tcp_ulp.c clears the socket's SOCK_SUPPORT_ZC flag right before attaching the ULP. I looked up who reads that flag in 6.8: io_uring/net.c, at the start of the IORING_OP_SEND_ZC and SENDMSG_ZC operations, does test_bit(SOCK_SUPPORT_ZC) and returns -EOPNOTSUPP if it is missing. So on a socket that passed through OpenSSL 3.0.13's hands, you cannot do zero-copy sends with io_uring, even though you are not doing TLS in the kernel.

Second, sk_psock_init() in net/core/skmsg.c checks inet_csk_has_ulp(sk) first thing and bails out with -EINVAL if set. BPF sockmap/sockhash goes through this function; a socket with a ULP attached cannot enter the map. A minority of production setups use this; but for that minority it means a silent "why won't this socket go into the map" day.

Third, every socket gets a tls_context allocation, the tls module holds one reference per socket (106 sockets, 106 references; rmmod only once all of them close), and the close goes through tls_sk_proto_close(), which on seeing TLS_BASE skips the TLS cleanup, puts sk_prot back, calls TCP's own close and frees the context. Not a measurable cost, but not "nothing changed" either.

Is there no upside? The rationale the comment implies is convenience: with the ULP attached from the start, the second step happens immediately when SSL_OP_ENABLE_KTLS is set. Yet the complaint in #19676 is exactly where that convenience broke: if SSL_set_fd() is called before the connection exists, TCP_ULP fails with -ENOTCONN, nobody notices because the return value is unread, then TLS_TX is silently refused and days pass wondering "why won't kTLS turn on". So the "harmless" line both attached needlessly and failed to attach when needed; the PR moved it back to its real job, inside ktls_start().

Really turning it on: a lab

Turning it on takes one line in nginx; F5's 2021 kTLS post gives the same recipe: ssl_conf_command Options KTLS;. That writes the SSL_OP_ENABLE_KTLS flag through OpenSSL's SSL_CONF layer; nginx's own documentation for ssl_conf_command adds the warning that "configuring OpenSSL directly might result in unexpected behavior". Not unfairly, as we will see shortly.

Without touching my server's nginx, I brought up the nginx-debug binary from a separate prefix: Options KTLS on at 9443, off at 9444, both with the same self-signed certificate and sendfile on, serving a 20 MiB file of random bytes. During each transfer, ss -tni; before and after, tls_stat:

### port 9443 (KTLS) tls=1.3
before: TxSw=0 RxSw=0
tcp-ulp-tls version: 1.3 cipher: aes-gcm-256 rxconf: none txconf: sw
after:  TxSw=1 RxSw=0

### port 9443 (KTLS) tls=1.2
before: TxSw=1 RxSw=0
tcp-ulp-tls version: 1.2 cipher: aes-gcm-256 rxconf: sw txconf: sw
after:  TxSw=2 RxSw=1

### port 9444 (off) tls=1.3 and 1.2
tcp-ulp-tls rxconf: none txconf: none
after:  counters unchanged

The ss line comes back full when it is really on: version, cipher and sw. The difference from the none/none I had been looking at for four days is obvious at a glance. What I did not expect: OpenSSL 3.0.13 moves only the transmit path into the kernel for TLS 1.3; the receive path moves for TLS 1.2. The source says so too: the kTLS block in tls13_enc.c skips on !(which & SSL3_CC_WRITE), the block in t1_enc.c tries both directions. Which is why "I turned on kTLS" is an incomplete sentence; you have to ask which version and which direction.

The system calls show the difference just as starkly. Same 20 MiB, TLS 1.3, the nginx worker under strace:

# 9443, KTLS on
setsockopt(3, SOL_TCP, TCP_ULP, [7564404], 4) = 0
setsockopt(3, SOL_TLS, TLS_TX, "\4\0034\0..."..., 56) = 0
sendmsg(3, ... iov_len=265 ..., msg_control=[{cmsg_level=SOL_TLS, cmsg_type=0x1, cmsg_data="\x16"}])   x2
write(3, "HTTP/1.1 200 OK\r\n"..., 248) = 248
sendfile(3, 12, [0] => [2097152], 2097152) = 2097152
... 10 sendfile calls in total, 2 MiB each

# 9444, off
write(3, "\27\3\3@\21\243\233,"..., 16406) = 16406
... 1280 write calls in total

On the off side, 1280 writes of 16406 bytes: 16384 bytes of plaintext plus a 5-byte header, a 16-byte tag and 1 byte of inner content type; that is the arithmetic of a TLS 1.3 record, and the @\21 in the header is 0x4011, the length field for a 16401-byte body. On the on side the key is handed to the kernel in a 56-byte crypto_info structure, followed by two sendmsg calls: these are the session tickets sent after the handshake, the path the kernel documentation calls a "control message"; cmsg_type=0x1 is TLS_SET_RECORD_TYPE, and 0x16 is record type 22, handshake. Plaintext goes in, the kernel encrypts. The close_notify at the end of the connection takes the same path, with type 21. Then the file goes out via sendfile() in 2 MiB pieces; 2 MiB has been the default of sendfile_max_chunk since nginx 1.21.4. The same release note carries the line "support for SSL_sendfile() when using OpenSSL 3.0"; the feature I am looking at today arrived with that note. nginx's debug log tells the same story in other words: BIO_get_ktls_send(): 1, then SSL to sendfile: @0 2097152 and SSL_sendfile: 2097152 lines. Grepping for those two strings is the verification method the F5 post recommends; I came to prefer the txconf: sw field in ss, because it does not require turning on debug logging.

On speed I have nothing to say. Transfers over loopback with a rate-limited curl were not designed to measure time. F5 says "up to 2x" and "nearly 30% without any specific tuning" for their own test; the distance between the two numbers is an admission of how much the gain depends on disk and load. Do not expect a number without measuring on your own workload.

What breaks if you turn it on under Ubuntu 24.04

After enabling Options KTLS, TLS 1.3's KeyUpdate message came to mind. The kernel documentation is clear on this: when a KeyUpdate is received, the receive path is paused until the new key is supplied via TLS_RX, reads in the meantime return EKEYEXPIRED; there is no pausing on the transmit side. But that paragraph is in 6.14's documentation; TLS 1.3 rekeying entered the kernel with the change titled "tls: implement rekey for TLS1.3", and the TlsTxRekeyOk counters have been in the documentation since that release. The tls_stat output on my 6.8 does not have those counters, and 6.8's do_tls_setsockopt_conf() answers a second TLS_TX attempt with -EBUSY, under the comment "Currently we don't support set crypto info more than one time".

So does OpenSSL 3.0.13 even try to give the kernel a second key? The K command of s_client sends a KeyUpdate that also asks the peer to update its own key. Same scenario against both ports: connect, K, one second later GET /index.html:

# 9444, kTLS off
KEYUPDATE
HTTP/1.1 200 OK
hello ktls

# 9443, kTLS on
KEYUPDATE
SSL routines:ssl3_get_record:decryption failed or bad record mac

On the off side both parties switch keys and carry on. On the on side, the server honors the client's request and sends its own KeyUpdate; that is the five-byte \30\0\0\1\0 going out via sendmsg in strace (type 24, length 1, "update_not_requested"). Then comes write(3, "HTTP/1.1 200 OK"), and there is no new TLS_TX call in between. The source confirms it: tls13_update_key() in 3.0.13 derives the new traffic key, writes it into the userspace cipher context and returns; the kernel never hears of it. The kernel keeps encrypting with the old key, the client expects the new one, and the first record is a "bad record mac". Connection over.

As a control I also tried s_client's lowercase k command, which sends a KeyUpdate that does not ask the peer to change keys. On 9443 the result was 200 OK: since the server's receive path stays in userspace for TLS 1.3, the client changing its own key is fine; the problem is the server being forced to change its own transmit key. The trigger in this experiment was an s_client command; I did not measure which real clients send that kind of KeyUpdate, so I will not say "it breaks in browser X". But I can say this: on Ubuntu 24.04's 3.0.13 + 6.8 pair, any client that sends a KeyUpdate of type update_requested to a kTLS-enabled server loses its connection; and before kernel 6.14, even if OpenSSL had tried to do the right thing, it would have been turned away with -EBUSY. Upstream looks different: in an issue closed in May 2026 (#31138), an OpenSSL maintainer writes that KeyUpdate with kTLS is expected to work and EKEYEXPIRED should never be seen; the library installs the new key the moment it reads the KeyUpdate. That is a claim about the record layer rewritten in 3.2, and I did not measure it. Where you stand on 24.04 is where I measured.

Questions to answer before turning it on

Ask these of your own setup; when I asked, most of the answers came out "don't".

Are you serving static files or a cache, and over which protocol? kTLS's gain comes from sendfile(), and on HTTP/2 that gain shrinks: fetching the same 20 MiB with curl --http2, nginx still used sendfile(), but because the http2_chunk_size default is 8k it was 2560 calls of 8 KiB, each preceded by a separate write() for the 9-byte DATA frame header; 5120 calls against HTTP/1.1's 10. proxied dynamic responses still go through the SSL_write path, and what is saved there, in the OpenSSL documentation's telling, is the data being encrypted straight into kernel memory instead of encrypted and then copied in, that is, one copy. Most of my 71 connections on 443 are requests from Cloudflare passing through to backends; the blocks that serve files are a minority.

Which library? What decides the KeyUpdate breakage is the libssl version; since 3.0.13 never tries to hand the kernel a second key, moving 24.04 to a newer HWE kernel changes nothing by itself. Below kernel 6.14 there is -EBUSY on top. If you have a FIPS-style requirement that "all crypto goes through the provider", the OpenSSL documentation states plainly that kTLS encrypts in the kernel and bypasses the provider.

Which cipher suite? 3.0.13's Linux branch moves only AES-GCM-128/256, AES-CCM-128 (except on TLS 1.3, with the note "broken on 5.x kernels") and ChaCha20-Poly1305 into the kernel; on an unsupported suite the library silently falls back to userspace. If you see txconf: none, that is the first place to look.

How will you verify? Not the debug log, but txconf in ss -tni output and the increase of TlsTxSw in /proc/net/tls_stat; if you moved the receive path too, with TLS 1.2, TlsDecryptError lives in the same file, put it on the dashboard as well. The counters are kept per namespace; if you look from inside a container you see your own namespace.

And if you are not going to turn it on, does the stamp bother you? It is gone on OpenSSL 3.0.19 / 3.5.5 / 3.6.1 and above. On Ubuntu 24.04 it is not; if you use neither io_uring zero-copy sends nor BPF sockmap, the price is one socket context and a misleading number in lsmod. If you do use them, now you know why.

The comment in five places

What stays with me from this is not the ss output but that comment. "This call doesn't change any functionality of the socket" sat in five files for four years, compiled into every release, applied to every socket opened through that library. In the library's own world it was true: the bytes OpenSSL sent and received did not change. In the kernel's world it flipped three flags; the comment did not say so, and neither did I know, until strace and tcp_ulp.c were side by side.

A code comment tells the truth of the layer it was written in. The way to learn the other layer's truth is to go to that layer's code, or to count system calls without looking at what the comment says. The ss line belongs to the same class: it was not lying, I was reading it in the language of the neighboring layer rather than its own. It said "TLS socket options can be spoken here"; I read "TLS is in the kernel". In this morning's ECN piece I had mistaken negotiation for marking; in the afternoon I mistook the stamp for encryption. The same trap twice in one day; at least the second time I got out faster.

DEV Community