DEV Community

asif naeem
asif naeem

Posted on Edited on

I Swapped a Live nginx onto Post-Quantum TLS 6 Times Under Load. Zero Errors, and the OpenSSL Upgrade Paid for ML-KEM.

TL;DR

  • Moving nginx from Ubuntu's OpenSSL 3.0 to OpenSSL 3.5 with X25519MLKEM768, live, under 500 new TLS handshakes per second, dropped zero requests across 6 live swaps, 6 rollbacks (3 regular, 3 fast) and 54 config reloads. Across every run in this phase, over 7 million full handshakes, the error count was zero.
  • CPU: ML-KEM adds 68 µs per handshake (+19%), but OpenSSL 3.5 is 77 µs cheaper (−18%) than 3.0 for the same classical handshake on Graviton. Net cost of the whole migration: −2%.
  • The one real hazard: a PQ config line is fatal on the old binary. Roll back the binary without the config and the rollback fails silently. Traffic looks fine, and you're still on the new build.

The inspiration

Last part of the series found that 72% of the reachable top-1,000 sites already accept X25519MLKEM768. Now, from SRE standpoint the next followup question would be what does it cost me to switch my own nginx? instead of a benchmark lab — in a real production, with traffic flowing, with a rollback plan.

For most teams "switching" doesn't mean a new nginx. It means putting a newer OpenSSL under the nginx they already run. Ubuntu 24.04 ships OpenSSL 3.0.13, which can't do ML-KEM; we need 3.5. So that's the experiment.

Setup

nginx OpenSSL Groups offered Cert
A 1.28.0 3.0.13 (Ubuntu, shared lib) X25519, P-256 ECDSA P-256
B 1.28.0, identical configure flags 3.5.2, compiled in (--with-openssl) X25519MLKEM768, X25519, P-256 ECDSA P-256

Same nginx source, same flags, same classical certificate. Only the OpenSSL and the ssl_ecdh_curve line differ. Both binaries share one install prefix, which is what makes a live swap possible.

The target is a c7g.large (2 vCPU Graviton3), the same instance as Parts 1–3. The load generator is a c7g.xlarge in the same AZ and cluster placement group, running a small open-loop Go client. Every request is a brand-new TCP connection and a full TLS 1.3 handshake: no keep-alive, no session resumption, and the session cache is off. That's the worst case for handshake cost. Go's default client offers X25519MLKEM768 first and X25519 second, like current browsers, so the same client negotiates PQ with B and falls back on A.

nginx logs the negotiated group ($ssl_curve) and the worker PID for every request, and the response carries the PID back. That makes every request attributable to a specific binary generation, which is how you can watch a swap happen request by request.

1. The live swap

nginx has a built-in procedure for replacing its own binary without dropping connections:

  1. USR2 — the master starts the new binary as a second master, with its own workers. Both generations accept connections.
  2. WINCH — the old workers finish what they're doing and exit.
  3. QUIT — the old master exits.

I ran A → B this way three times under 500 handshakes/s, each time with the PQ config going in alongside the new binary.

One swap run: throughput, latency, PQ share and memory over time

Reading the chart top to bottom:

  • Throughput never moves. 270,003 requests across the three swap runs, 0 errors.
  • You can see the PQ cost live. The median request steps from about 1.57 ms to 1.67 ms the moment B takes over, and steps back after the rollback.
  • The handover is gradual. For the ~6 s between USR2 and WINCH, about half the handshakes are PQ, because old and new workers both accept. During a swap, a fleet serves both stacks for a moment; plan your dashboards accordingly.
  • Memory roughly doubles during the overlap (30 → ~69 MB here), because two full sets of workers are running. On this 2-worker box that's trivial. With worker_processes auto on a 64-core host, budget for roughly workers × per-worker memory extra for the length of the overlap. (The figure sums each process's resident memory, so it overstates real usage a little: binary pages are shared.)

In the 3 seconds after every signal (USR2, WINCH, QUIT, across all trials), p99 latency stayed within 0.1 ms of the quiet baseline.

2. The rollback trap

Here's the finding I didn't expect, and it showed up before any load test ran:

nginx: [emerg] SSL_CTX_set1_curves_list("X25519MLKEM768:X25519:prime256v1") failed
       (SSL: error:0A080106:SSL routines::passed invalid argument:group 'X25519MLKEM768' cannot be set)
Enter fullscreen mode Exit fullscreen mode

That's the old binary (OpenSSL 3.0) reading the new config. It doesn't skip the unknown group; the whole config fails to load.

Each swap run included a naive rollback: put the old binary back, leave the PQ config in place, send USR2. In all three trials, the new master hit that [emerg], exited, and the existing master carried on serving. No errors, no latency blip, no change in the PQ share. In the chart, it's the lone usr2 line at ~93 s where nothing happens.

That sounds safe, and for traffic it is. But think about what the operator sees: they ran the rollback, nothing broke, and the dashboards look normal. They are still on the new build. The only evidence is an [emerg] line in error.log, and a master PID that didn't change.

Two rollbacks that do work, both tested three times under load with zero errors:

  • Fast rollback, while the old master is still alive (after WINCH, before QUIT): send HUP to the old master, which restarts its workers with its in-memory binary and config, then QUIT the new master. The config on disk is irrelevant.
  • Late rollback, after QUIT: revert the config first, run nginx -t with the old binary, then swap the binary and USR2.

3. Reloads

Config reloads (nginx -s reload) are the most common production operation, so I ran nine of them, 30 s apart, under 500 handshakes/s: three trials per arm, 54 reloads in all. Zero errors. In the 3 s after each reload, p99 stayed within 0.1 ms of the quiet baseline on both arms. PQ doesn't change reload behaviour.

4. CPU: what a handshake costs

This needed two attempts (more on that below). The second measured CPU used by nginx's own processes only, at four connection rates, three trials each, in shuffled order.

nginx CPU vs connection rate, and CPU per handshake, for the three arms

arm CPU per full handshake implied ceiling on 2 vCPU
A: OpenSSL 3.0, classical 429 µs ~4,700 handshakes/s
B: OpenSSL 3.5, classical 352 µs ~5,700 handshakes/s
B: OpenSSL 3.5, X25519MLKEM768 419 µs ~4,800 handshakes/s

The per-handshake cost is flat from 1,000 to 4,000 handshakes/s, and trial-to-trial spread is under 10 µs. Each figure covers everything nginx does for that request (accept, TLS, HTTP, logging), not only the cryptography. The ceilings are extrapolated: the highest measured rate, 4,000/s, used 1.73 of the 2 cores on arm A with no errors.

Pulled apart:

  • Upgrading OpenSSL 3.0 → 3.5 alone: −77 µs per handshake (−18%). OpenSSL 3.5 does the same classical handshake noticeably faster on Graviton.
  • Adding X25519MLKEM768 on 3.5: +68 µs (+19%). That's the price of the extra key encapsulation, plus a ClientHello and ServerHello each about 1 kB bigger.
  • Net, A → B with PQ: −2%. If you're on a distro OpenSSL today, the upgrade you need for PQ roughly pays for PQ.

One caveat about 4,000/s: for A and B-with-PQ (87–89% host CPU) latency starts to rise, with the median handshake at 1.2–1.3 ms versus 0.7–0.9 ms at lower rates. B-classical's p99 at that rate averages 95 ms across its three trials, so at least one trial had a latency spike. I'm treating 4,000/s as the edge of saturation and not comparing tail latency there.

5. Build, footprint, memory

A (OpenSSL 3.0, shared) B (OpenSSL 3.5, static)
Build time on c7g.large 20 s 122 s (OpenSSL compile)
Stripped binary 1.2 MB 7.6 MB
TLS code incl. shared libs 6.6 MB 7.6 MB (+16%)
Idle RSS, all processes 23.5 MB 24.1 MB
Per-worker RSS under load 14.0 MB 14.6 MB
RSS growth over 10 min / 300k handshakes 2.1 MB/h 2.3 MB/h

The binary looks 6× bigger, but A loads its OpenSSL from shared libraries; counted fairly, B is 16% bigger. There's no leak: both arms grew about 2 MB/h, which is allocator warm-up, not a slope that continues. One small oddity: after a live swap or several reloads, the master process settles around 8 MB versus 2 MB after a cold start. It's a fixed ~6 MB per host, not per worker, and it stays flat.

The instrument failed twice first (again)

A recurring theme of this series is that the measuring tool hits its limits before the system under test does.

First, rollback detection. My swap script originally declared success when nginx renamed its pid file to .oldbin. nginx does that before the new binary even parses its config, so my script reported the failed naive rollback as a success. The fix: a swap only counts as successful if, a moment later, two masters are actually running. That's also the check to put in your own runbook.

Second, CPU. The first CPU ladder sampled whole-host CPU, one 60 s run per step, in a fixed order. At 1,000 handshakes/s it showed PQ as cheaper than classical, and one run drifted from 34% to 17% CPU at a constant request rate. Kernel networking, sshd copying the previous run's logs, and ordering effects were all mixed in. The rerun counted only nginx's processes (from /proc/<pid>/stat), used three shuffled trials, and excluded the ramp-up seconds. The same 1,000 handshakes/s step that looked backwards in host CPU (A 19.5%, B-classical 26.8%) is cleanly ordered in nginx CPU (A 0.41 cores, B-classical 0.35).

The runbook

If I were doing this on a production fleet:

  1. Build B in CI, with the same configure flags as your current nginx (nginx -V prints them). Budget about 2 minutes for the OpenSSL compile.
  2. Check headroom: CPU is roughly a wash if you're coming from a distro OpenSSL 3.0. Memory needs room for a second full set of workers during the swap.
  3. Test the new pair before touching the running server: new-nginx -t -c new.conf.
  4. Swap binary and config together: USR2 → verify (two masters running, $ssl_curve showing X25519MLKEM768 in the access log) → WINCH.
  5. Keep the old master alive until you're satisfied. While it lives, rollback is HUP old + QUIT new, with no config dependency.
  6. After QUIT, rollback means config first, then nginx -t with the old binary, then the binary.
  7. Alert on [emerg] in error.log and on master PID changes. A failed rollback is otherwise invisible.

Method and limits

  • nginx 1.28.0; OpenSSL 3.0.13 (Ubuntu 24.04 package) vs 3.5.2 (upstream, default build options, statically linked); c7g.large target, c7g.xlarge load generator, same AZ, cluster placement group.
  • Every request is a full TLS 1.3 handshake with a 3-byte response. That isolates the handshake path and is the worst case; with resumption and keep-alive, real traffic costs much less per request.
  • ECDSA P-256 certificate on both arms. This is the KEM-first migration you can ship today; PQ certificates are a separate problem (Part 4).
  • The A/B CPU comparison is distro-built OpenSSL 3.0 vs upstream-built 3.5. Part of the 18% could come from build options rather than the version; it is the comparison an operator actually faces.
  • One instance type, one region, one load shape. Per-handshake µs will differ on x86 and on bigger instances.
  • Everything for this phase (Terraform, build scripts, load generator, scenario orchestration, analysis, and per-run results) is in its own repo, pqc-bench-nginx-swap, commit 9065728. The whole matrix costs under $1 to reproduce.

Correction and follow-up

A reader made two points in the comments, and both deserve data rather than a reply.

First: zero errors is the right headline and the least interesting number. A correct USR2 swap should drop nothing. What actually changes is the handshake, so the numbers to watch are TLS handshake p99 and the resumption ratio. That's fair. I led with errors because "does the swap drop traffic?" was the question I kept being asked, but this post should have been clearer about where the cost goes.

Second: separate the two populations. Long-lived clients mostly resume their TLS sessions; cold connections pay for a full handshake. Every request in the runs above was a cold, full handshake (session cache and tickets off), so the post measured only one population.

So I ran the split. Same three arms, same c7g.large, same AZ. Session tickets are on in every run, so the server config is identical, and only the client changes: 0%, 50% or 90% of connections present a session ticket. Each mix ran 3 trials at 2,000 connections/s, in shuffled order. Every request records whether it actually resumed, and nginx's $ssl_session_reused confirms it from the server side. Trial-to-trial spread was under 4 µs, and there were zero errors across all 27 runs.

CPU per handshake vs resumption share, and handshake p99 for cold vs resumed connections

CPU: resumption doesn't avoid ML-KEM

arm µs per cold handshake µs per resumed handshake saved by resuming
A: OpenSSL 3.0, classical 493 418 75 (15%)
B: OpenSSL 3.5, classical 398 335 63 (16%)
B: OpenSSL 3.5, X25519MLKEM768 466 403 63 (14%)

(Cold and resumed costs are separated by fitting a line across the three mixes for each arm.)

  • ML-KEM costs +68 µs per handshake, cold or resumed. That matches the CPU ladder above to the microsecond, a nice independent replication.
  • The reason: TLS 1.3 resumption, in the mode browsers and Go clients actually use (psk_dhe_ke), still does a fresh key exchange. nginx's log shows 100% of resumed handshakes on the PQ arm still negotiated X25519MLKEM768. Resumption skips the certificate and the server's signature, not the KEM.
  • So ML-KEM's relative cost is slightly higher for long-lived clients: +20% on a resumed handshake vs +17% on a cold one. Resumed clients do still show the cost; it's the same 68 µs, just a bigger share of a cheaper handshake.
  • On OpenSSL 3.5, resumption saves almost exactly what ML-KEM costs. A resumed PQ handshake (403 µs) costs about the same as a cold classical one (398 µs).

Latency: here the reader was right

cold handshake p50 / p99 resumed handshake p50 / p99
A: OpenSSL 3.0, classical 0.81 / 1.13 ms 0.65 / 0.92 ms
B: OpenSSL 3.5, classical 0.71 / 0.97 ms 0.58 / 0.83 ms
B: OpenSSL 3.5, X25519MLKEM768 0.92 / 1.29 ms 0.76 / 1.07 ms

(Cold from the 0%-resumed runs, resumed from the 90% runs; same-AZ round-trip times, so these are almost entirely server and client compute.)

  • The migration lowers CPU but raises the tail. Going from A to B with PQ, CPU per cold handshake falls 5% (493 → 466 µs), but cold handshake p99 rises 14% (1.13 → 1.29 ms).
  • On the same OpenSSL 3.5 build, ML-KEM adds about 0.33 ms to cold p99 (+34%) and about 0.24 ms to resumed p99 (+28%). CPU averages hide this. The extra key encapsulation and the ~1 kB larger ClientHello and ServerHello sit directly on the handshake's critical path, and p99 shows it first.

What this changes

The body of this post said CPU is "roughly a wash". That holds, but on its own it was the wrong conclusion to stop at:

  • The swap is free in CPU, not in latency. Watch TLS handshake p99, not error rate or average CPU.
  • Resumption doesn't shield long-lived clients from the cost. They pay the same +68 µs of CPU. It's a larger share of a cheaper handshake, and their p99 still rises by about 0.24 ms.
  • Session tickets have a cost of their own. With tickets on, a full handshake cost 49–68 µs more than in the tickets-off ladder at the same rate, because the server now encrypts and sends session tickets on every full handshake. That's worth knowing before attributing a CPU change to PQ.
  • Runbook addition: track handshake p99 split by $ssl_session_reused, alongside the share of connections using X25519MLKEM768 ($ssl_curve), before and after the swap.

Caveats as before: one instance type (2 vCPU Graviton3), same AZ, ECDSA P-256 certificate (its signature is cheap, so resumption saves less here than it would with RSA), a 3-byte response, and a Go client. Over a real WAN, the extra bytes also cost round-trip time; this setup measures compute, not distance.

Data and scripts are in results/7b/. Thanks to the commenter: this was the right question to ask.


Next: the SLH-DSA measurements, once the monitor turns green.

Top comments (6)

Collapse
 
aiapikey profile image
HAOHAOnovai •

Zero errors is the right headline and the least interesting number: a correct swap should produce zero errors. What actually moves is the handshake. ML-KEM makes the initial flight larger and adds CPU per full handshake, so the metrics worth watching are p99 TLS handshake time and the resumption ratio - not error rate.

Split those two populations before concluding the swap was free: long-lived clients mostly resume and will barely show it, while cold connections pay the full cost and show it in the tail first. Your part 1 and 2 already separate cost per handshake from cost per request, which is the distinction most write-ups skip.

Collapse
 
asif_naeem_bd5842be56bfcd profile image
asif naeem •

Fair point, and thanks for reading parts 1 and 2 closely. Zero errors is the floor, not the finding: I led with it because “does the swap drop traffic?” was the question I got from a reader, but you’re right that the handshake is where the cost moves.

One clarification on what was measured: every request in this run was a cold full handshake (session cache and tickets off, new connection each time), so it’s entirely the population you describe as showing it first. In that population, handshake p99 went from ~0.85 ms to ~0.97 ms at 500 conn/s (same AZ, 3 trials each), and ML-KEM cost +68 µs CPU per full handshake. That’s the same +19% the post reports, offset by OpenSSL 3.5 being cheaper than 3.0.

On resumption, one nuance: TLS 1.3 resumption in the mode browsers and OpenSSL actually use (psk_dhe_ke) still does a fresh key exchange, so resumed handshakes pay the ML-KEM cost too. What they skip is the certificate and the signature. My guess is that ML-KEM’s relative overhead is higher on resumed handshakes, not lower. I haven’t measured that split yet, and my next trial would be, cold vs resumed, with handshake p99 and resumption ratio tracked separately.

Collapse
 
asif_naeem_bd5842be56bfcd profile image
asif naeem •

hi 于侃,
I ran the split. Same three arms, session tickets on, with the client resuming 0%, 50% or 90% of connections, 3 trials each at 2,000 conn/s:

ML-KEM costs +68 µs CPU per handshake whether cold or resumed. Resumption doesn’t avoid it: every resumed handshake on the PQ arm still negotiated X25519MLKEM768 (psk_dhe_ke), confirmed in nginx’s log. Relative overhead is +17% cold vs +20% resumed, because resumption skips the cert and signature but not the key exchange.
You were right about the tail. Going from OpenSSL 3.0 to 3.5+PQ lowers CPU per cold handshake (493 → 466 µs) but raises handshake p99 (1.13 → 1.29 ms). On the same 3.5 build, PQ adds ~0.33 ms to cold p99 and ~0.24 ms to resumed p99.
So “the swap was free” holds for CPU, not latency. And long-lived clients don’t escape it by resuming: they pay the same +68 µs, it’s just a bigger slice of a cheaper handshake.

Data and scripts: pqc-bench-nginx-swap, results/7b/.

Thread Thread
 
aiapikey profile image
HAOHAOnovai •

Those numbers reframe it well: CPU down ~27 us per cold handshake but p99 up ~160 us means you did not remove the cost, you moved it from CPU to the tail. That is a much better trade than it reads as, because CPU was the scarce resource during the swap.

The number I would watch at fleet scale is the resumed-handshake one. +68 us per resumed handshake sounds free per request, but resumption is the common path, so at 10k resumed handshakes/s across a pool that is ~0.68 core-seconds of added service time every second - a capacity question rather than a latency one, and it shows up first as queueing on the busiest nginx, not as a higher p99 anywhere obvious.

Did you measure a resumption ratio alongside the p99s? Without it, the +68 us is hard to compare across traffic mixes.

Thread Thread
 
asif_naeem_bd5842be56bfcd profile image
asif naeem •

hi 于侃,
Yes, measured both sides: the client recorded per request whether it actually resumed, and nginx logged $ssl_session_reused. They agree at 0.0 / 0.50 / 0.90 for the three mixes. The useful part is that the ML-KEM delta doesn’t depend on the mix: fitting cold and resumed cost separately gives +68 µs for each. So across traffic mixes the added CPU is just 68 µs × handshakes/s, whatever the ratio. Your 10k/s → ~0.68 core figure is exactly the right way to size it.

One refinement on “moved from CPU to the tail”: those are two changes stacked. The OpenSSL 3.0 → 3.5 upgrade alone lowered CPU and p99 (cold: 493 → 398 µs, 1.13 → 0.97 ms). Adding ML-KEM on top raised both again (+68 µs, +0.33 ms p99). So the upgrade bought headroom, and PQ spent part of it in CPU and more than all of it in tail latency. If you’re stuck on a distro OpenSSL 3.0, the net looks like a trade; on a fleet already on 3.5, it’s a pure cost.

Agree it shows up as queueing first. In the CPU ladder, at ~87–89% CPU the median handshake rose from ~0.8 ms to ~1.25 ms before anything else looked wrong. At fleet scale I’d alert on handshakes/s per node (split by $ssl_session_reused) against that node’s measured ceiling, not on p99 alone. Keep-alive makes request rate a poor proxy, since handshake rate is what costs.

Thread Thread
 
aiapikey profile image
HAOHAOnovai •

Your reframe splits the change into two moves with opposite signs: from your numbers CPU per cold handshake goes 493 → 466 µs (-5.5%) while cold p99 goes 1.13 → 1.29 ms (+14%). So the upgrade buys capacity and PQ spends tail. A fleet that was CPU-capped gains room; a fleet that was tail-capped loses.

Which is why I would alert on the knee rather than a fixed percentage of CPU: find the handshakes/s per vCPU where p99 crosses your SLA, before and after PQ, and alert at a fraction of that measured knee. On your ladder it moves below 87-89%, so a static threshold fires after the fact - the exact failure you are describing.

One more field worth logging per node: the $ssl_session_reused ratio, because it changes what the knee means as much as load does.