Your Guide to the Prerequisites for Running Nodes on Oasis (Part 1): Hardware and Cloud Providers

There are various ways to support a blockchain protocol. For example, tech-savvy users sometimes prefer to run nodes. Now, if you are planning to do so, specific hardware and software prerequisites must be in place. In this 4-part series, I will discuss them in detail for running nodes on Oasis. The first part covers hardware and cloud providers.

Hardware Requirements

As outlined in the introduction to Oasis nodes, the protocol consists of several distinct node types.

• Consensus validator or non-validator node
• Sapphire compute or client node
• Emerald compute or client node
• Cipher compute or client node

In this section, I will focus on the minimum and recommended system hardware requirements. Anyone planning to run more than one paratime (Sapphire/Emerald/Cipher) on a single node will need more resources.

If you configure a system with fewer resources than recommended, you run the risk of being under-provisioned. This can cause proposer node timeouts and synchronization delays. If your node exhausts its memory or storage, the node process is forcefully killed, which can result in state corruption. As a result, you risk losing stake and getting barred from participating in committees.

CPUs

Consensus validator or non-validator node

• Minimum: 2.0 GHz x86-64 CPU with AES instruction set support
• Recommended: 2.0 GHz x86-64 CPU with 2 cores/vCPUs with AES instruction set and Advanced Vector Extensions 2 (AVX2) support

Emerald compute node and all client nodes

• Minimum: same as above
• Recommended: 2.0 GHz x86-64 CPU with 4 cores/vCPUs with AES instruction set and AVX2 support

Sapphire/Cipher ParaTime compute node

• Minimum: 2.0 GHz x86-64 CPU with AES instruction set and Intel SGX support
• Recommended: 2.0 GHz x86-64 CPU with 2 cores/vCPUs with AES instruction set, Intel SGX, and AVX2 support

Notes:

1. If you are running regular workloads on your node, the minimum resources will suffice. However, if you plan heavier workloads, like a paratime client node behind a public Web3 gateway, you will need more cores/vCPUs.
2. The reason you need AES instruction set support is that it is required by Deoxys-II-256-128, a Misuse-Resistant Authenticated Encryption (MRAE) algorithm, for encrypting paratime state.
3. The reason you need AVX2 support is to enable faster Ed25519 signature verification. This will speed up node sync.
4. You need Intel SGX support only when you are running paratime compute nodes (Sapphire/Cipher). Check out Intel's comprehensive list of processors that support Intel SGX, and match it with Sapphire and Cipher requirements before proceeding.

Memory

Consensus validator or non-validator node

• Minimum: 6 GB of ECC RAM
• Recommended: 8 GB of ECC RAM

Each Emerald/Sapphire/Cipher compute or client node

• Minimum: 12 GB of ECC RAM
• Recommended: 20 GB of ECC RAM

In most scenarios, the nodes, even with less than the minimum amount of memory, will run regular workloads smoothly. To cover the more resource-intensive time points, such as the initial state sync, BadgerDB migration when upgrading a node to a new major version of the Oasis Core, generating storage checkpoints with BadgerDB, and periodic BadgerDB compactions, your nodes will need more memory than usual.

Storage

Consensus validator or non-validator node

• Minimum: 400 GB of SSD or NVMe fast storage
• Recommended: 700 GB of SSD or NVMe fast storage

Emerald compute or client node (in addition to the consensus storage requirements)

• Minimum: 400 GB of SSD or NVMe fast storage
• Recommended: 700 GB of SSD or NVMe fast storage

Sapphire/Cipher compute or client node (in addition to the consensus storage requirements)

• Minimum: 200 GB of SSD or NVMe fast storage
• Recommended: 300 GB of SSD or NVMe fast storage

Notes:

1. Both consensus and paratime states are stored in an embedded BadgerDB database. As BadgerDB is designed to run on SSDs, it is highly recommended not to run a node that stores data on HDDs.
2. Both consensus and paratime states accumulate over time. The rate of storage growth varies by transaction count on the network and the paratimes. For example, a consensus non-validator node on the mainnet accumulated 280 GB of consensus state in approximately one year since the Cobalt upgrade, and 32 GB of consensus state in approximately one month since the Damask upgrade. Another example would be an Emerald client node on the mainnet, which additionally accumulated 260 GB of paratime state in approximately five months since Emerald's launch, and 25 GB of paratime state in approximately one month since the Damask upgrade.
3. Dump and restore upgrades, such as the Damask and Cobalt upgrades, include state wipes that free up node storage. If you need to access historical state, you can run a separate archive node. Also note, the latest is Eden
4. It is recommended not to keep a complete state from the genesis onward unless you are running a node to preserve historical state. This will also reduce storage requirements for both consensus and paratime states, as well as improve the overall latency of the storage-related operations. To enable storage reduction, you should check out consensus pruning and paratime pruning.

Network

Consensus validator node and all paratime compute nodes

• Minimum: 200 Mbps internet connection with low latency
• Recommended: 1 Gbps internet connection with low latency

Regular workloads use relatively little traffic. The higher bandwidth is recommended for bursts of transactions, when a node with too little bandwidth can time out.

Cloud Providers

If you plan to run a Sapphire or Cipher compute node, a crucial step is verifying processor compatibility and checking the status of Intel SGX support (refer to Note #4 in the CPUs section).

Limitations

You will come across several bare-metal or dedicated server providers using Intel processors that support SGX. However, you should also be aware of potential limitations.

• BIOS configuration: Some providers restrict users from accessing or modifying BIOS settings, which might be essential to correctly configure Intel SGX.
• Outdated firmware: SGX requires the latest firmware to function correctly. If the system provider is behind on updates, this is a problem.
• Lack of SGX-specific offerings: Some providers do not advertise or specifically offer SGX-enabled servers. Users need to know specifically that their server is SGX-enabled, as this feature is non-negotiable for running some Oasis nodes.
• Limited support: Sometimes even when an SGX-enabled server is available, the provider's support team might lack the expertise to handle SGX-specific issues or configurations.
• Hardware provisioning: Use of keys, such as SGX sealing keys, involves them being bound to hardware to encrypt the data of an instance within an Intel SGX enclave. If the host of the instance is changed, the encrypted data cannot be decrypted.

Known providers

1. Alibaba Cloud
   Product: (ECS) Bare Metal Instances
   Documentation: Build an SGX confidential computing environment

2. Azure
   Product: Some Dedicated Host SKUs
   Documentation: Solutions on Azure for Intel SGX

3. Gcore
   Product: Bare Metal
   Documentation: Computing with Intel SGX

4. IBM Cloud
   Product: Virtual Private Cloud (VPC)
   Documentation: Confidential computing with SGX for VPC

5. OVH
   Product: Bare Metal servers
   Documentation: SGX for Confidential Computing

6. PhoenixNAP
   Product: Bare Metal Cloud
   Documentation: What is Intel SGX and What are the Benefits?

7. Vultr
   Product: Bare Metal
   Documentation: Intel SGX development on Vultr

Note: Provider offerings change, so the list above may not reflect each provider's current SGX support. So, it is recommended to check out the provider's own documentation and confirm Intel SGX availability with them directly.
If any cloud or dedicated server providers that actively support Intel SGX or Intel TDX are not in this list, or if you come across updated information about the providers listed above, you can let the Oasis team know by creating an issue on GitHub with the additional details.

This concludes part 1 of the prerequisites series. In the next part, I will discuss stake requirements and how to install the Oasis node.

• For technical specs, APIs, architecture, and integration guides, the Oasis documentation is your starting point.
• For direct support on specific issues, the Oasis engineering team and other node operators are available in the node-operators channel on the official Discord.