PQC-Bench Part 4: SLH-DSA over TLS failed

TL;DR

• SLH-DSA signatures are ~17 kB. That’s bigger than TCP’s initial congestion window (14.6 kB), so a cert chain with SLH-DSA would need an extra round-trip to deliver. I wanted to measure that effect at WAN RTT.

• We can generate SLH-DSA keys and sign things with them in OpenSSL 3.5.2. We cannot serve an SLH-DSA cert over TLS 1.3 — not with vanilla OpenSSL, and not with oqs-provider 0.12.0 either.

• The TLS layer is a gap. SLH-DSA is production-ready for our signing API. It is not production-ready for our TLS terminator. Here’s exactly where the gap lives.

Context

In Phase 3 of pqc-bench (Part 3 here) we measured the handshake latency of ML-DSA-65, ECDSA, and RSA-2048 across three AWS regions. Recap: cert-chain-size differences are statistically invisible at WAN RTT when the whole chain fits inside TCP’s initial congestion window of 14.6 kB. ML-DSA-65’s 3.3 kB signature → 5–6 kB chain → fits comfortably in one round-trip. No measurable penalty.

Phase 4 was going to extend that: pick the one PQ signature algorithm whose chain doesn’t fit, measure the extra round-trip explicitly, and close the story. SLH-DSA-SHA2-128f was the obvious choice. Its 17,088-byte signature puts the chain around 20 kB — the first PQ signature at NIST level 1 that would force a second RTT.

What was supposed to be an evening of config bumping turned into three distinct failures that each taught me something about how PQ TLS support actually ships in the real world.

Attempt 1:

OpenSSL 3.5.0 lists SLH-DSA but can’t generate keys

My Phase 1–3 AMIs were built on OpenSSL 3.5.0, released May 2026. The algorithm registry looked promising:

$ /opt/openssl-3.5/bin/openssl list -public-key-algorithms | grep SLH
Name: OpenSSL SLH-DSA-SHA2-128f implementation
IDs: { 2.16.840.1.101.3.4.3.21, id-slh-dsa-sha2-128f, SLH-DSA-SHA2-128f }

But:

$ openssl genpkey -algorithm SLH-DSA-SHA2-128f -out slh.key
Error: unsupported algorithm

OIDs registered, operations not implemented. OpenSSL 3.5.2 (Aug 2026) fills this in.

Next attempt: Rebuild the AMI, bump the pin. ~20 minutes.

Attempt 2:

3.5.2 keygen works, nginx still can’t serve the cert

With 3.5.2 baked, keygen succeeds:

$ openssl genpkey -algorithm SLH-DSA-SHA2-128f -out /tmp/t.key && echo OK
OK
$ openssl req -new -x509 -key /tmp/t.key -out /tmp/t.crt -days 365 -subj "/CN=test"
$ wc -c /tmp/t.crt
23556 /tmp/t.crt

~23 kB cert chain — exactly as predicted, well past initcwnd. The crypto all works. Hand this cert to nginx:

nginx
ssl_certificate /tmp/t.crt;
ssl_certificate_key /tmp/t.key;
ssl_protocols TLSv1.3;
ssl_conf_command SignatureAlgorithms SLH-DSA-SHA2-128f;
$ sudo nginx-pq -t
nginx: [emerg] SSL_CTX_use_certificate("/tmp/t.crt") failed
(SSL: error:0A0000F7:SSL routines::unknown certificate type)

nginx can’t load an SLH-DSA cert at all. The error comes from OpenSSL’s TLS layer, not nginx. Let's dig into why:

$ openssl list -tls-signature-algorithms | tr ':' '\n' | grep -i slh
(empty)
$ openssl list -tls-signature-algorithms | tr ':' '\n' | grep -i dsa
mldsa44
mldsa65
mldsa87

OpenSSL 3.5.2 has SLH-DSA at the X.509 and signing layers, but not at the TLS layer. ML-DSA shows up as a TLS 1.3 signature algorithm. SLH-DSA doesn’t. The TLS code path has no idea what to do with an SLH-DSA cert.

The pattern recognition kicks in: this is what oqs-provider from the Open Quantum Safe project was built for. Install it alongside OpenSSL, get the missing TLS integration -> measure -> ship.
That was my next plan of action.

Attempt 3: oqs-provider doesn’t fix it either

Rebuilt from source: liboqs (shared), oqs-provider v0.12.0-dev linked against OpenSSL 3.5.2, both installed into /opt/openssl-3.5.2/. openssl.cnf activates it at init:

`[provider_sect]
default = default_sect
oqsprovider = oqsprovider_sect

[default_sect]
activate = 1

[oqsprovider_sect]
activate = 1`

Smoke test:

$ openssl list -providers
Providers:
default
name: OpenSSL Default Provider
version: 3.5.2
status: active
oqsprovider
name: OpenSSL OQS Provider
version: 0.12.0-dev
status: active

Both active. Good start. Then:

$ openssl list -signature-algorithms | grep -i slh
{ 2.16.840.1.101.3.4.3.21, SLH-DSA-SHA2-128f } @ default
(... all other SLH-DSA variants, all @ default ...)

Every SLH-DSA entry shows @ default. Zero @ oqsprovider. Which means oqs-provider v0.12 has stopped claiming SLH-DSA — it defers to OpenSSL’s default provider now that 3.5.2 has native SLH-DSA keygen and signing.

OK, the registration moved. Does the TLS layer work now? Let's try to serve the cert:

$ openssl s_server -key /tmp/t.key -cert /tmp/t.crt -accept 127.0.0.1:8443 -tls1_3
Using default temp DH parameters
error setting certificate
error:0A0000F7:SSL routines:ssl_set_cert:unknown certificate type:ssl/ssl_rsa.c:269

Identical error to attempt 1. oqs-provider ceded SLH-DSA to the default provider, but OpenSSL’s default provider still has only two of the three layers it needs. Nobody is wiring up SLH-DSA at the TLS layer in October 2026.

The three layers

TLS support for a signature algorithm isn’t one thing. It’s three layers that have to arrive together:

Cryptographic primitives — keygen, sign, verify. openssl genpkey -algorithm X works.
X.509 encoding — the algorithm’s OID is understood when parsing cert ASN.1. openssl req -new -x509 -key X works.
TLS protocol integration — the algorithm is advertised in ClientHello’s signature_algorithms extension, accepted by SSL_CTX_use_certificate, selected during negotiation, carried in CertificateVerify.

For ML-DSA, all three are in OpenSSL 3.5.2’s default provider. You can genkey, you can sign a cert, nginx can serve it, clients negotiate it. Full stack.

For SLH-DSA, OpenSSL 3.5.2 has layers 1 and 2. Nobody has layer 3 yet. Not OpenSSL’s default. Not oqs-provider 0.12. The error isn’t an obscure config nit — it’s ssl_set_cert saying “I don’t know what this is.”

What this means if you’re planning PQ TLS

Use ML-DSA if your TLS stack is OpenSSL ≥ 3.5.0 or anything built on it (nginx, Envoy, HAProxy with modern crypto, Go ≥ 1.24). It works today, chain fits in initcwnd for the common security levels (ML-DSA-65 at NIST level 3), no RTT penalty vs classical.

Do not plan an SLH-DSA deployment at the TLS termination layer today. The ecosystem gap is real and precise: X.509 cert works, signing API works, TLS handshake fails. If you have a signing use case that fits (JWT, firmware, package signatures, code signing), SLH-DSA is production-ready for that right now — just not through your TLS listener.

Monitoring the gap close

The simplest check from any machine with OpenSSL 3.5.2 and oqs-provider installed:

bash
openssl list -tls-signature-algorithms | tr ':' '\n' | grep slh

The day this returns slhdsasha2128f (or similar), SLH-DSA at TLS has arrived, probably in a new oqs-provider release that re-registers the TLS codepoints after OpenSSL added the keygen. I’ll be checking monthly. When it changes, Phase 5 of pqc-bench measures what I set out to measure tonight.

What’s in the repo

pqc-bench-slh-dsa on GitHub has all the exercise frozen in a branch: the user-data templates that build the full oqs-provider stack, the acceptance tests that fail cleanly, and the full logs of all three failure modes. If you’re staring at a 0A0000F7 error from your own PQ experiment, you’ll find the exact same thing in those logs.