Mustafa ERBAY

Posted on Aug 8 • Originally published at mustafaerbay.com.tr

Deploying an ML-KEM Hybrid TLS Pilot with Performance Measurement

#sistemmimarisi #software

The potential widespread adoption of quantum computers threatens our current public-key cryptography algorithms, posing a serious risk to our critical infrastructures. In response to this threat, post-quantum cryptography (PQC) algorithms like ML-KEM (Module-Lattice-based Key Encapsulation Mechanism) are being tested with hybrid approaches to future-proof our TLS connections. This article will cover the practical steps and technical details of deploying an ML-KEM hybrid TLS pilot with performance measurement.

Deploying a hybrid TLS pilot requires much more than just integrating new algorithms; carefully analyzing compatibility with existing systems, performance impacts, and reliability factors is critical. Especially in enterprise environments, this transition process must be managed incrementally and in a controlled manner. Otherwise, unexpected outages or security vulnerabilities could arise.

The Rise of Post-Quantum Cryptography and ML-KEM

The potential of quantum computers to break current asymmetric encryption algorithms like RSA and ECC using Shor's algorithm is a serious concern for long-term information security. This threat brings to light the "harvest now, decrypt later" scenario, especially for data that needs to remain confidential and have a long lifespan. For this reason, NIST (National Institute of Standards and Technology) initiated a standardization process to develop quantum-safe algorithms.

Among the algorithms selected in NIST's third round of standardization and standardized as FIPS 203, ML-KEM (formerly CRYSTALS-Kyber) stands out as a prominent solution for key encapsulation mechanisms. ML-KEM is based on lattice-based cryptography principles and, unlike existing classical algorithms, is assumed to be resistant to known quantum computer attacks. This feature makes it one of the cornerstones of future secure communication protocols.

ℹ️ What is ML-KEM?

ML-KEM (Module-Lattice-Based Key-Encapsulation Mechanism) is a lattice-based Key Encapsulation Mechanism (KEM) developed to counter the risk of quantum computers breaking existing encryption algorithms. It is one of the main algorithms standardized as FIPS 203 in NIST's post-quantum cryptography standardization process.

ML-KEM's design balances efficiency and security. However, as a new algorithm, it necessitates detailed examination of its integration with existing systems and its performance characteristics. Therefore, comprehensive pilot implementations and measurements are essential before transitioning to production environments.

Why is a Hybrid TLS Architecture Necessary?

The transition to post-quantum cryptography cannot be a "big-bang" operation done all at once. Since it's uncertain when and how quickly existing systems will be broken by quantum threats, a hybrid TLS architecture is proposed as a transition strategy. Hybrid TLS combines both classical (e.g., ECDHE) and PQC (e.g., ML-KEM) key exchange algorithms in the same TLS handshake.

The primary goal of this approach is to "minimize security risk." This means if an unexpected vulnerability is found in the PQC algorithm, the classical algorithm will still provide security; or if the classical algorithm is broken by quantum computers, the PQC component will step in to offer protection. This "two-layered" security creates a safety net against the unknowns that may arise during the transition.

Hybrid TLS has the potential to minimize existing compatibility issues, especially in large and complex infrastructures. Since classical algorithms are still widely supported, it plays a critical role in this period when PQC algorithms are new and not yet fully adopted by all client/server software. However, the added complexity and potential performance costs introduced by the hybrid approach must also be considered.

Environment Preparation for Pilot Implementation

To deploy an ML-KEM hybrid TLS pilot, an appropriate test environment must first be prepared. This environment should be a copy of the production environment or accurately simulate it. Core requirements include PQC-enabled versions of cryptographic libraries like OpenSSL and compatible web servers (Nginx, Apache).

Required Software and Versions

For the pilot, the following updated or PQC-enabled software versions are typically used:

OpenSSL: OpenSSL 3.5.0 (released April 2025) and later versions have native support for NIST standardized PQC algorithms such as ML-KEM, ML-DSA, and SLH-DSA, and use hybrid PQC key exchange (e.g., X25519MLKEM768) by default. For older versions like OpenSSL 3.0.x or 3.1.x, PQC support can be added via the oqs-provider module of the Open Quantum Safe (OQS) project.
Web Server: Versions of popular servers like Nginx or Apache compiled with or dynamically linked to the PQC-enabled OpenSSL version.
Client: A PQC-enabled curl or a custom-written Python/Go client for testing. Modern versions of curl compiled with a PQC-enabled TLS library support hybrid key exchange with the --curves option. As of August 2025, browsers like Firefox and Chrome also support hybrid PQC key exchanges like X25519MLKEM768 by default.

💡 OpenSSL Compilation

OpenSSL 3.5.0 and later versions natively and by default support PQC algorithms, so they do not require a special enable-pqc flag. If you are using an older 3.x version of OpenSSL and want to add PQC support via oqs-provider, this would involve installing oqs-provider and enabling it in the OpenSSL configuration, rather than compiling OpenSSL from source. Compiling to a custom directory and pointing applications to this specific build is a safer approach than modifying system-wide OpenSSL libraries.

Test Environment Topology

A simple pilot environment typically consists of two main components: a server and a client. The web server and PQC-enabled OpenSSL run on the server side, while a testing tool using the same OpenSSL version runs on the client side. In more complex scenarios, a load balancer or proxy might also be included.

An example environment setup might include the following steps:

Operating System Installation: A clean Linux distribution (Ubuntu, Debian, CentOS) is preferred.
Dependency Installation: Required build tools and libraries (gcc, make, perl, zlib-devel, etc.) are installed.
OpenSSL PQC Compilation: The official OpenSSL source code is downloaded and compiled with commands like ./config --prefix=/opt/openssl-pqc shared zlib. For OpenSSL 3.5 and later, PQC support is included by default.
Web Server Compilation/Configuration: Compiling Nginx with PQC-enabled OpenSSL or configuring an existing Nginx to use PQC-enabled OpenSSL libraries.

# Compile OpenSSL 3.5.0 or later (Example)
# This compilation will include PQC support by default.
git clone https://github.com/openssl/openssl.git
cd openssl
# Use OpenSSL 3.5.0 or a more recent stable version
git checkout openssl-3.5.0 # Or a more current LTS version
./config --prefix=/opt/openssl-pqc shared zlib
make -j$(nproc)
sudo make install

# Compile Nginx with PQC-enabled OpenSSL (Example)
# Use a current stable version of Nginx (e.g., 1.26.x)
wget http://nginx.org/download/nginx-1.26.0.tar.gz
tar -xzvf nginx-1.26.0.tar.gz
cd nginx-1.26.0
./configure \
    --with-openssl=/opt/openssl-pqc \
    --with-http_ssl_module \
    --prefix=/opt/nginx-pqc
make -j$(nproc)
sudo make install

⚠️ Warning About Modifying System Libraries

The compilation steps above install OpenSSL and Nginx into custom directories. However, modifying system-wide libraries or software managed by the existing package manager can be risky in terms of system stability and security updates. Before making such changes in production environments, comprehensive backups should be taken, dry-run tests performed, and a rollback plan established.

These steps form the foundation of the test environment, providing a platform where ML-KEM hybrid TLS can be run. Carefully documenting each step in this process is important for troubleshooting potential issues and ensuring reproducibility.

ML-KEM Hybrid TLS Configuration and Test Scenarios

To enable hybrid TLS, the web server (e.g., Nginx) and client (e.g., curl) configurations must be set correctly. This step aims to ensure that both classical and post-quantum key exchange algorithms work together. On the Nginx side, PQC algorithms are specified using directives like ssl_conf_command Ciphersuites and ssl_conf_command Curves for TLS 1.3 in the SSL/TLS configuration.

Hybrid TLS Configuration with Nginx

In the Nginx configuration, it's important to define cipher suites and key exchange groups that include PQC algorithms. An example nginx.conf file might contain a structure like this:

server {
    listen 443 ssl;
    server_name your_domain.com;

    ssl_certificate /path/to/your/certificate.crt;
    ssl_certificate_key /path/to/your/private.key;

    # Only secure protocols
    ssl_protocols TLSv1.3 TLSv1.2; 

    # Hybrid key exchange groups for TLS 1.3
    # X25519MLKEM768 is a standard hybrid group supported by OpenSSL 3.5+.
    # P-256 (prime256v1) is a classical elliptic curve group.
    ssl_conf_command Curves X25519MLKEM768:P-256; # Or X25519MLKEM768:prime256v1

    # Cipher suites for TLS 1.3
    ssl_conf_command Ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256;

    # Cipher suites for TLS 1.2 and earlier
    # This list should be kept current for Nginx compiled with OpenSSL 3.5+.
    ssl_ciphers "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256";

    ssl_prefer_server_ciphers on; # Server preference for TLS 1.2

    location / {
        root /usr/share/nginx/html;
        index index.html index.htm;
    }
}

This configuration ensures that Nginx offers both classical and ML-KEM-supported key exchange groups and cipher suites. The client then selects the appropriate one from this list. The ssl_prefer_server_ciphers on directive encourages the server to use its preferred cipher suite for TLS 1.2 and earlier versions.

Test Scenarios and Client Configuration

Testing different scenarios in the pilot implementation is important for understanding performance and compatibility:

Classical TLS Only: Connect using only classical algorithms to establish a performance baseline.
PQC TLS Only (Single Algorithm): Connect using only PQC algorithms like ML-KEM. This shows the pure performance impact of PQC.
Hybrid TLS: A scenario where both classical and PQC algorithms are used together. This simulates a real-world transition.

On the client side, a PQC-enabled curl command or OpenSSL's s_client tool can be used.

# Hybrid connection test with PQC-enabled OpenSSL s_client (example)
# X25519MLKEM768 is the hybrid key exchange group included with OpenSSL 3.5+.
/opt/openssl-pqc/bin/openssl s_client -connect your_domain.com:443 -curves X25519MLKEM768:P-256 -msg -state -tls1_3

# Or with PQC-enabled curl (curl must be compiled with PQC support)
# curl must be compiled with a PQC-enabled TLS library (e.g., OpenSSL 3.5+).
/opt/curl-pqc/bin/curl -v --curves X25519MLKEM768 https://your_domain.com

These tests verify both the success of the key exchange and show the algorithms used during the handshake. The -msg and -state parameters in the s_client tool are very useful for monitoring the details of the TLS handshake.

This flowchart summarizes the main steps of the pilot. Each step requires careful planning and validation. Especially the steps of measuring and analyzing performance will be decisive for the decision to move hybrid TLS to a production environment.

Defining and Collecting Performance Metrics

One of the most critical steps in the ML-KEM hybrid TLS pilot is accurately measuring performance impacts. New cryptographic algorithms have the potential to introduce overhead on processor (CPU) and memory usage. Therefore, the pilot's objective should go beyond functional verification and also provide a comparative analysis against a performance baseline.

Key Metrics to Measure

The primary metrics to focus on for performance measurements include:

TLS Handshake Latency: The time taken to establish a TLS connection. This time may increase due to larger key sizes and more complex mathematical operations of PQC algorithms.
CPU Usage: CPU consumption on both the server and client sides, especially during key exchange and encryption/decryption operations.
Memory Usage: The amount of memory PQC algorithms require to store keys and other cryptographic data.
Bandwidth Usage: The amount of data transferred during the handshake, as PQC keys and certificates are generally larger.
Throughput: The number of connections or amount of data that can be processed in a given time.

Performance Measurement Tools

Various tools can be used to collect these metrics:

openssl s_client: Shows TLS handshake times and used algorithms in detail.
time command: Used to measure the execution time of general commands or scripts.
iperf3: Ideal for measuring network bandwidth and latency. Helps evaluate data transfer performance over the TLS layer.
wrk or ab (ApacheBench): Used to apply load tests to HTTP/HTTPS servers, measuring concurrent connections and throughput.
top, htop, vmstat, sar: For real-time or periodic monitoring of CPU, memory, and disk I/O usage on the server.
Custom Scripts: Written in Python or Go, automating specific scenarios and recording metrics.

⚠️ Measurement Accuracy

When performing performance measurements, ensure that environmental factors (network congestion, impact of other processes) are minimized. Taking repeated measurements and evaluating their averages provides more reliable results. Conducting realistic, repeatable, and controlled tests is vital.

Data Analysis and Comparison

The collected data should highlight the differences between classical TLS and hybrid TLS. For example, if the TLS handshake time for critical API endpoints in an ERP system of a manufacturing company increases from 50ms to 100ms, this could directly impact user experience. Similarly, if the web server's CPU usage increases by 10%, this could mean additional hardware costs or less concurrent connection capacity. Analyses should clearly articulate such trade-offs.

Potential Challenges and Solution Approaches

It is possible to encounter various technical challenges when deploying an ML-KEM hybrid TLS pilot. These challenges typically arise in the areas of compatibility, performance, certificate management, and debugging. Anticipating these issues and developing proactive solution approaches are key to the pilot's success.

Compatibility Issues

While PQC algorithms like ML-KEM have been standardized, widespread software and hardware support will take time. This can lead to compatibility issues, especially in environments using older systems or specialized hardware. For example, situations may arise where an old proxy server or hardware load balancer does not understand PQC cipher suites or key exchange groups.

Solution Approach: Conduct multi-layered tests. Compatibility tests should be performed with different client/server versions and intermediary devices. If necessary, specialized proxies or software load balancers can be used to route PQC traffic.

Performance Overhead

Larger key sizes and more complex cryptographic operations can create additional load on the CPU and memory. This can lead to performance bottlenecks, especially in high-traffic services or resource-constrained devices.

Solution Approach: Determine the differences between baseline and hybrid TLS through comprehensive performance tests. If the performance degradation is unacceptable, consider switching from higher security PQC algorithms (e.g., ML-KEM-768) to lower security ones (e.g., ML-KEM-512) or hardware upgrades. Additionally, optimization techniques such as TLS session caching can be used.

Certificate Management

Hybrid TLS can affect not only key exchange algorithms but also certificates. The generation and management of PQC-signed certificates or hybrid certificates can add complexity to the existing PKI (Public Key Infrastructure) structure.

Solution Approach: Create a PQC-enabled Certificate Authority (CA) or investigate PQC support for existing CAs. Additionally, certificate validation processes can be accelerated with technologies like OCSP Stapling. Automation of certificate renewal and distribution processes will help manage this complexity.

Debugging and Observability

It is natural to encounter unexpected errors during the pilot of a new technology. TLS handshake errors, cipher suite mismatches, or performance issues can be difficult to root cause.

Solution Approach: Use detailed logging and observability tools. OpenSSL's debug modes, Nginx's access and error logs, and system tools like strace play a critical role in diagnosing problems. Furthermore, distributed tracing can help identify the source of the problem by monitoring each stage of the connection flow. For example, issues with TLS handshake getting stuck at a certain stage might be due to an old proxy misinterpreting PQC extensions. Such problems can be identified through detailed logs and packet capture tools.

These challenges require careful planning, continuous monitoring, and flexible solution approaches for the successful deployment of ML-KEM hybrid TLS.

Conclusion

The process of deploying an ML-KEM hybrid TLS pilot with performance measurement is an important first step in the transition to post-quantum cryptography. This process is not just a technological integration but also a comprehensive effort in risk management and performance optimization. As the threat of quantum computers becomes increasingly concrete, preparing our existing cryptographic infrastructure for the future is a strategic imperative.

The environment preparation, configuration steps, collection of detailed performance metrics, and solution approaches to potential challenges discussed in this guide will ensure that this transition is carried out in a controlled and informed manner. The hybrid TLS architecture provides valuable experience for future full PQC integration while protecting our existing systems. It should be remembered that such major technological changes can always bring unexpected problems; however, with proper planning and continuous observation, these problems can be overcome. The next step will be to use the data obtained from these pilot studies to create a broader deployment strategy.

DEV Community