What Is the OpenSSL Engine API? Integrating Cloud HSMs and YubiKeys with OpenSSL

#openssl #security

What Is the OpenSSL Engine API?

While OpenSSL implements fundamental cryptographic operations such as encryption and signing on its own, it also provides a plugin-like mechanism called the Engine API that allows these operations to be delegated to external hardware. By using the Engine API, cryptographic operations provided by cloud-based HSMs (Hardware Security Modules) or other external hardware can be called transparently through OpenSSL.

By delegating cryptographic operations to secure hardware such as HSMs, it becomes possible to perform operations like signing while keeping the private key stored securely on the hardware — all while still using OpenSSL.

Examples of Engine API Usage

One example of the Engine API in action is the pkcs11 engine plugin. This plugin enables OpenSSL to access cryptographic devices that implement the PKCS#11 interface.

Google has published the Google PKCS #11 Cloud KMS Library, which allows Google Cloud HSMs to be operated via PKCS#11. By using this library, encryption and signing operations can be executed on Google Cloud's HSMs.

Similarly, Yubico has published YKCS11, which enables YubiKey hardware to be operated via PKCS#11. Through this, OpenSSL can invoke operations that use asymmetric private keys stored on the YubiKey hardware.

Benefits of Externalizing Signing and Other Operations via the Engine API

By using the OpenSSL Engine API, it becomes possible to perform cryptographic operations such as signing on FIPS 140-2 Level 3 compliant hardware like Google Cloud HSM. As a result, private keys never leave the HSM, significantly reducing the risk of key leakage.

A real-world example of a serious security incident caused by key leakage is the discovery in 2022 that Nvidia's code signing certificates were being used to sign malware.

Because incidents involving the leak of code signing keys have occurred repeatedly, an industry rule was established requiring that, as of June 1, 2023, private keys used for code signing must be stored on FIPS 140-2 Level 2 compliant hardware¹². The Engine API has become an essential means of meeting these industry requirements.