How to encrypt the NVS volume on the ESP32

There has been a lot of discussion around embedded device security during the last few years especially after well-publicized DDoS incidents involving armies of hijacked IoT devices. The demand for higher levels of security has put pressure on manufacturers and software providers to adopt and support modern security protocols in order to mitigate the relevant risks especially given the widening spread of IoT devices.

An IoT device may embed various security-related artifacts in order to enable the use of relevant protocols, for example certificates for communicating securely with remote servers (mqtt, http etc.). It is really easy for a bad actor with physical access to the device to snatch binary images from the device's storage and search for such pieces of text. The possible leak of a security certificate may lead to escalated attacks into other parts of the infrastructure especially if the relevant permissions are incorrect or relaxed enough. In order to avoid that, we need to ensure that the device's storage is encrypted and that no sensitive information embedded in the firware image or elsewhere can be read even with physical access to the device.

In this article, we will demonstrate how to encrypt the non-volatile storage of the ESP32. The ESP32 is a very popular choice for building embedded solutions considering the SoC's capabilities (dual core, wifi, bluetooth), its low price (< 5$) and its comprehensive SDK framework including a tcp/ip stack, http server/client (incl. TLS support), OTA firmware updates etc.

In general, encryption on the ESP32 is supported on the hardware level so as to prevent the recovery of (most) SPI flash contents using physical readouts. The ESP32 has two basic types of partitions: app which contain application-related artifacts such as the device firmware and data which contain arbitrary user data. The encryption of app-type partitions is fairly straight-forward and well-documented. The process roughly consists of building the firmware with support for encryption, flashing the device and leaving the rest to the bootloader. Partitions marked as data however are not handled automatically by the bootloader and require a different process with respect to encryption.

The non-volatile storage of the ESP32 is a data-type partition that uses a portion of the underlying flash over SPI and is typically used for storing key-value pairs. These can include, for example, unique device identification strings, wifi configuration data (incl. passwords), device-specific security certificates etc., in other words stuff that we would like to keep private from curious eyes. The ESP32 supports NVS encryption but, as mentioned before, the process is a little bit more involved.

NVS supports two flavours of encryption:

• runtime-encryption whereby the application itself generates the key and encrypts/decrypts data on the fly, and
• build-time encryption whereby the nvs volume is pre-encrypted and flashed to the device

In the runtime-encryption method, the application generates a key using the corresponding esp-idf function and uses this key in order to encrypt/decrypt data in the nvs volume at runtime. These can be, for example, wifi or other passwords that may be known only at runtime and not beforehand.

In the build-time encryption method, the NVS partition containing all the necessary key-value pairs is prepared and encrypted for downloading to the device. This method can be used in cases where the NVS data are known at compile time -- example of such data include unique device IDs, device-specific security certificates etc.

In both cases, a separate partition is necessary for storing the nvs encryption key. ESP-IDF provides a partition subtype for this purpose (type data and subtype nvs_keys) and handles its encryption transparently via the bootloader.

The use case that we'd like to demonstrate here is the baking of pre-existing data (unique device ID, security certificates) into the device at compile time. OK, so let's go through this procedure step-by-step by means of an example.

First, we need a custom partition table which can be configured using idf.py menuconfig and selecting "Custom Partition Table (CSV)" under the "Partition Table" menu (CONFIG_PARTITION_TABLE_CUSTOM=y and CONFIG_PARTITION_TABLE_FILENAME="partitions.csv"). Our partitions.csv file looks like this:

# ESP-IDF Partition Table
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,       ,  0x4000,
otadata,  data, ota,       ,  0x2000,
phy_init, data, phy,       ,  0x1000,
factory,  0,    0,         , 1M,
ota_0,    0,    ota_0,    , 1M,
ota_1,    0,    ota_1,    , 1M,
nvs_key,  data, nvs_keys,         , 0x1000, encrypted

This partition table supports 3 app partitions (ESP's standard OTA scheme with 1 factory and 2 OTA partitions), one 16KB nvs partition and one 4KB nvs_keys partition. We have only specified partition sizes -- the offsets are calculated automatically by the tools. The encrypted flag of the nvs_key partition instructs the bootloader to automatically encrypt the contents, since we're going to be storing the nvs encryption keys there. (Of course it'd be great if we could do that for the nvs partition as well, but this feature is not supported unfortunately as the existence of this article demonstrates...)

Next, we are going to prepare our nvs image locally by specifying its contents using the CSV format as follows:

# NVS csv file
key,type,encoding,value
device_id,data,string,a_unique_value
cert,file,string,./path/to/certificate.pem.crt
pvkey,file,string,./path/to/private.pem.key

The actual image (nvs.bin) can be generated from the csv file (nvs.csv) using the partition generation tool (given a specified $IDF_PATH):

$ $IDF_PATH/components/nvs_flash/nvs_partition_generator/nvs_partition_gen.py generate nvs.csv nvs.bin 0x4000 // not encrypted

However, the resulting image nvs.bin will be unencrypted. If instead of the generate command we use the encrypt command, the resulting image will be encrypted and the tool will output a second image (nvs_keys.bin) with the contents of the encryption key:

$ $IDF_PATH/components/nvs_flash/nvs_partition_generator/nvs_partition_gen.py encrypt nvs.csv encrypted_nvs.bin 0x4000 --keygen --keyfile nvs_keys.bin

These two images can now be flashed to the device using esptool.py (part of the esp-idf distribution):

$ esptool.py -p PORT --before default_reset --after no_reset write_flash 0xa000 encrypted_nvs.bin

where PORT is the serial comm device address (something like /dev/cu.usbserial-0001) and encrypted_nvs.bin is the image file that we generated in our previous step. The flash location address 0xa000 can be discovered either by inspecting the esp32 serial output (where the partitions are printed out) or by using the gen_esp32part.py utility on the project's partition image (e.g. gen_esp32part.py build/partition_table/partition-table.bin).

The nvs_keys image also needs to be downloaded to the device:

$ esptool.py -p PORT --before default_reset --after no_reset write_flash 0x320000 nvs_keys.bin

We're almost done. On the application side, we now need to initialize the secure nvs volume like so:

esp_err_t nvs_secure_initialize() {
    static const char *nvs_tag = "nvs";
    esp_err_t err = ESP_OK;

    // 1. find partition with nvs_keys
    const esp_partition_t *partition = esp_partition_find_first(ESP_PARTITION_TYPE_DATA,
                                                                ESP_PARTITION_SUBTYPE_DATA_NVS_KEYS,
                                                                "nvs_key");
    if (partition == NULL) {
        ESP_LOGE(nvs_tag, "Could not locate nvs_key partition. Aborting.");
        return ESP_FAIL;
    }

    // 2. read nvs_keys from key partition
    nvs_sec_cfg_t cfg;
    if (ESP_OK != (err = nvs_flash_read_security_cfg(partition, &cfg))) {
        ESP_LOGE(nvs_tag, "Failed to read nvs keys (rc=0x%x)", err);
        return err;
    }

    // 3. initialize nvs partition
    if (ESP_OK != (err = nvs_flash_secure_init(&cfg))) {
        ESP_LOGE(nvs_tag, "failed to initialize nvs partition (err=0x%x). Aborting.", err);
        return err;
    };

    return err;
}

void app_main() {
    esp_err_t err = nvs_secure_initialize();
    if (err != ESP_OK) {
        ESP_LOGE("main", "Failed to initialize nvs (rc=0x%x). Halting.", err);
        while(1) { vTaskDelay(100); }
    }

    // rest of application code goes here
    // ...
}

Once the app is built and flashed (using idf.py encrypted-flash), we're good to go with our encrypted NVS volume and our IoT device can now be safely deployed to the field.

Comments

[Palok Biswas]

I have a separate NVS partition for device credentials. Everything worked as usual after following your instructions, however, nvs_open_from_partition() failed to open the partition. I initialized the partition using nvs_flash_secure_init_partition() and it gives me an error of namespace not existing. Any idea why that is happening?

[Palok Biswas]

nvs_flash_secure_init_partition is working now but nvs_get_str() is unable to find the key value that was in my nvs partition. Works fine without the encryption

[Kyriakos Kentzoglanakis]

Are you still facing a problem? I am using nvs_open before calling nvs_get_str which returns the stored string successfully (if that helps).

[Alexandre Beaulieu]

Thank you for this article. It was very helpful. Just a note that when developing with the encrypted flash in development mode, the NVS key must be written with the --encrypt flag otherwise nvs_flash_read_security_cfg will return 0x1117 (ESP_ERR_NVS_CORRUPT_KEY_PART).

In the example, this would be:
esptool.py -p PORT --before default_reset --after no_reset write_flash --encrypt 0x320000 nvs_keys.bin

[Achraf Boussaada]

So I tried to create a custom nvs partition and write data to it and flash it to the chip and then programmatically initialize it and read data from it. That worked fine.
Now with the esptool.py I tried to encrypt the partition generate its keys and flash it to the chip. Following the documentation (docs.espressif.com/projects/esp-id...) and your post I tried to read data from the encrypted partition. The problem I'm having is that the data I flashed seems not be found. I used nvs_get_stats() to get Infos about the entries of the new partition but I always get entries count is 1. This didn't happen before when I tried it with the unencrypted partition. Also, intialilizing and openening the encrypted partition without nvs_flash_secure_init_partition() gives me the same result and doesn't through an error about encryption keys needed or something.

[Achraf Boussaada]

After further inverstigation nvs_flash_read_security_cfg() is giving me ESP_ERR_NVS_CORRUPT_KEY_PART

[Achraf Boussaada]

when you're flashing the nvs_keys.bin you gave 0x320000 as a location address. How did you determine that?

[Kyriakos Kentzoglanakis]

The address of a particular partition can be discovered either by inspecting the esp32 serial output (where the partitions are printed out) or by applying the esp-idf's gen_esp32part.py utility on the project's partition image (e.g. gen_esp32part.py build/partition_table/partition-table.bin)

[Achraf Boussaada]

Thanks for the reply. I just realised that you need to create another partition to store the keys, and the partition should be flaged as encrypted. I'm sorry if this seems obvious, but I'm new to this, is there no security risk doing this?

[Kyriakos Kentzoglanakis]

The partition that stores the encryption keys for the nvs partition is itself encrypted by the device (handled by the bootloader -- more info here. The reason for the existence of the keys partition is that the esp32 does not (yet?) handle the encryption of an nvs partition transparently, so we have to do it ourselves.

[Achraf Boussaada]

Thanks. Final question about the keys partition. According to the documentation the partition needs to have a encrypted flag. Did you add it manually when creating the partition from the CSV file?

[MarcioCarleto]

I've deleted the lines "cert,file,string,./path/to/certificate.pem.crt" and "pvkey,file,string,./path/to/private.pem.key" because I don´t have the .crt or .key in my application, it generated the encrypted_nvs.bin and nvs_keys.bin files but I don´t know what is the encrypted_nvs.bin address of my application I should use (yours is 0xa000 right?). I've attached my partition-table offsets when I build my project, can you please help on this?

[MarcioCarleto]

I've deleted the lines "cert,file,string,./path/to/certificate.pem.crt" and "pvkey,file,string,./path/to/private.pem.key" because I don´t have the .crt or .key in my application, it generated the encrypted_nvs.bin and nvs_keys.bin files but I don´t know what is the encrypted_nvs.bin address of my application I should use (yours is 0xa000 right?). I've attached my partition-table offsets when I build my project, can you please help on this?

[mocasrhey]

i flashed the partition and it executes nvs_open with success but nvs_get_str returns ESP_ERR_NVS_NOT_FOUND, do you have any idea of what might be the problem?

[Kyriakos Kentzoglanakis]

My guess would be that the key does not exist. Did you try to write a key/value pair to nvs first before you attempt to retrieve the key's value?