DEV Community

Achyuta Das
Achyuta Das

Posted on

OpenBao HA Cluster with Auto-Unseal using TPM

#vault #cluster #deployment

Guide to deploy a 3-node Raft cluster with TLS everywhere, SoftHSM2 PKCS#11 auto-unseal, per-node vTPM pin sealing, and keepalived VIP for leader failover.

Node Reference

Role IP
Node 1 (bootstrap leader) 192.168.1.10
Node 2 (follower) 192.168.1.11
Node 3 (follower) 192.168.1.12
VIP (floats to leader) 192.168.1.13
  • OS: Ubuntu 24.04 LTS
  • OpenBao: v2.5.1 (HSM build)
  • Storage: Integrated Raft
  • Auto-unseal: SoftHSM2 PKCS#11 (AES-256-GCM) + vTPM-sealed pin per node

How It Works

All three nodes share one SoftHSM2 token containing a single AES-256 key. Raft bootstrap requires this — the leader encrypts a join challenge with the seal key and the joining node must decrypt it with the same key.

Each node's SoftHSM userpin is sealed to that node's own vTPM. On every start, a wrapper script calls tpm2_unseal to retrieve the pin, passes it to OpenBao via BAO_HSM_PIN, and the node auto-unseals. No human intervention. If the vTPM is gone (VM migrated without vTPM state), the node cannot start — that's the hardware-binding guarantee.

Node 1 (192.168.1.10)        Node 2 (192.168.1.11)        Node 3 (192.168.1.12)
┌──────────────────────┐     ┌──────────────────────┐     ┌──────────────────────┐
│ SoftHSM token        │────▶│ SoftHSM token (copy) │────▶│ SoftHSM token (copy) │
│ AES-256 key          │     │ AES-256 key          │     │ AES-256 key          │
│ pin → Node 1 vTPM    │     │ pin → Node 2 vTPM    │     │ pin → Node 3 vTPM    │
└──────────────────────┘     └──────────────────────┘     └──────────────────────┘
                               VIP: 192.168.1.13
                         (floats to current Raft leader)
Enter fullscreen mode Exit fullscreen mode

Prerequisites

  • 3 nodes (VMs or bare-metal) running Ubuntu 24.04 LTS with network connectivity between them.
  • Each node must have a TPM 2.0 — either a physical TPM or a virtual TPM (vTPM) provided by the hypervisor. Verify with: 
  ls -la /dev/tpm0 /dev/tpmrm0
Enter fullscreen mode Exit fullscreen mode

Both devices must exist. If they don't, enable vTPM in your hypervisor settings or confirm the physical TPM is enabled in BIOS/UEFI.

  • openssl installed on whichever machine you use to generate certs (can be any of the nodes or your workstation).
  • SSH access to all three nodes with sudo.

Phase 1 — All 3 Nodes: System Setup, Packages, OpenBao Install

Run everything in this phase on all three nodes.

1.1 Update and Install Dependencies 

sudo apt-get update && sudo apt-get upgrade -y

sudo apt-get install -y \
  tpm2-tools \
  tpm2-abrmd \
  softhsm2 \
  opensc \
  keepalived \
  jq \
  curl
Enter fullscreen mode Exit fullscreen mode

1.2 Install OpenBao HSM Binary 

curl -fsSL "https://github.com/openbao/openbao/releases/download/v2.5.1/bao-hsm_2.5.1_Linux_x86_64.tar.gz" \
  -o /tmp/bao-hsm.tar.gz

tar -xzf /tmp/bao-hsm.tar.gz -C /tmp/
sudo mv /tmp/bao /usr/local/bin/bao
sudo chmod +x /usr/local/bin/bao
bao version
Enter fullscreen mode Exit fullscreen mode

1.3 Create User, Directories, TPM Access 

sudo useradd --system --home /etc/openbao --shell /bin/false openbao

sudo mkdir -p \
  /etc/openbao/tpm2 \
  /etc/openbao/softhsm2/tokens \
  /opt/openbao/data \
  /opt/openbao/tls

sudo chown -R openbao:openbao /etc/openbao /opt/openbao
sudo chmod 750 /etc/openbao /opt/openbao/data
sudo chmod 700 /etc/openbao/tpm2 /etc/openbao/softhsm2 /etc/openbao/softhsm2/tokens

# SoftHSM config
sudo tee /etc/openbao/softhsm2.conf > /dev/null <<'EOF'
directories.tokendir = /etc/openbao/softhsm2/tokens
objectstore.backend = file
log.level = INFO
EOF
sudo chown openbao:openbao /etc/openbao/softhsm2.conf
sudo chmod 0640 /etc/openbao/softhsm2.conf

# TPM access
sudo systemctl enable --now tpm2-abrmd
sudo usermod -aG tss openbao
Enter fullscreen mode Exit fullscreen mode

1.4 Generate and Distribute TLS Certificates

Run this on one machine (any node or your workstation). The same self-signed cert is used on all nodes — it includes every node IP, the VIP, and localhost as SANs.

CLUSTER_NAME="openbao-cluster"
WORKDIR="$(mktemp -d)"

cat > "${WORKDIR}/openssl.cnf" <<EOF
[req]
default_bits       = 2048
encrypt_key        = no
default_md         = sha256
prompt             = no
utf8               = yes
distinguished_name = req_distinguished_name
req_extensions     = v3_req

[req_distinguished_name]
CN = ${CLUSTER_NAME}

[v3_req]
basicConstraints     = CA:FALSE
keyUsage             = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage     = serverAuth, clientAuth
subjectAltName       = @alt_names

[alt_names]
DNS.1 = localhost
DNS.2 = ${CLUSTER_NAME}
IP.1  = 192.168.1.10
IP.2  = 192.168.1.11
IP.3  = 192.168.1.12
IP.4  = 192.168.1.13
IP.5  = 127.0.0.1
EOF

# Generate private key
openssl genrsa -out "${WORKDIR}/openbao-key.pem" 2048

# Generate self-signed certificate (valid 10 years)
openssl req -new -x509 -days 3650 \
  -key "${WORKDIR}/openbao-key.pem" \
  -out "${WORKDIR}/openbao-cert.pem" \
  -config "${WORKDIR}/openssl.cnf" \
  -extensions v3_req

# Verify SANs
openssl x509 -in "${WORKDIR}/openbao-cert.pem" -noout -text | grep -A1 "Subject Alternative Name"
Enter fullscreen mode Exit fullscreen mode

Distribute to all 3 nodes:

for NODE_IP in 192.168.1.10 192.168.1.11 192.168.1.12; do
  scp "${WORKDIR}/openbao-cert.pem" "${WORKDIR}/openbao-key.pem" <user>@${NODE_IP}:/tmp/
done
Enter fullscreen mode Exit fullscreen mode

Then on each node, move the certs into place:

sudo mv /tmp/openbao-cert.pem /opt/openbao/tls/openbao-cert.pem
sudo mv /tmp/openbao-key.pem  /opt/openbao/tls/openbao-key.pem

sudo chown openbao:openbao /opt/openbao/tls/openbao-cert.pem /opt/openbao/tls/openbao-key.pem
sudo chmod 0644 /opt/openbao/tls/openbao-cert.pem
sudo chmod 0600 /opt/openbao/tls/openbao-key.pem
Enter fullscreen mode Exit fullscreen mode

1.5 Create Start Wrapper Script

This script runs on every service start. It unseals the pin from the local vTPM and passes it to OpenBao.

sudo tee /usr/local/bin/openbao-start > /dev/null <<'SCRIPT'
#!/bin/bash
set -euo pipefail

TMPDIR="$(mktemp -d)"

tpm2_createprimary -C o -G rsa2048 -c "$TMPDIR/primary.ctx" >/dev/null 2>&1
tpm2_load \
  -C "$TMPDIR/primary.ctx" \
  -u /etc/openbao/tpm2/pin.pub \
  -r /etc/openbao/tpm2/pin.priv \
  -c "$TMPDIR/pin.ctx" >/dev/null 2>&1

BAO_HSM_PIN="$(tpm2_unseal -c "$TMPDIR/pin.ctx")"
rm -rf "$TMPDIR"

export BAO_HSM_PIN
export SOFTHSM2_CONF=/etc/openbao/softhsm2.conf

exec /usr/local/bin/bao server -config=/etc/openbao/config.hcl
SCRIPT

sudo chmod +x /usr/local/bin/openbao-start
Enter fullscreen mode Exit fullscreen mode

1.6 Systemd Unit 

sudo tee /etc/systemd/system/openbao.service > /dev/null <<'EOF'
[Unit]
Description=OpenBao Secret Management Service
Documentation=https://openbao.org/
Requires=network-online.target tpm2-abrmd.service
After=network-online.target tpm2-abrmd.service
ConditionFileNotEmpty=/etc/openbao/config.hcl

[Service]
User=openbao
Group=openbao
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=no
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
AmbientCapabilities=CAP_IPC_LOCK
NoNewPrivileges=yes
ReadWritePaths=/etc/openbao/softhsm2
ExecStart=/usr/local/bin/openbao-start
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable openbao
Enter fullscreen mode Exit fullscreen mode

ReadWritePaths=/etc/openbao/softhsm2 lets SoftHSM write lock files despite ProtectSystem=full.
PrivateDevices=no is required for TPM access via tpm2-abrmd.

Phase 2 — All 3 Nodes: OpenBao Config and Keepalived

2.1 OpenBao Configuration

Run on each node, replacing NODE_IP, NODE_ID with that node's values:

Node NODE_ID NODE_IP
1 node1 192.168.1.10
2 node2 192.168.1.11
3 node3 192.168.1.12 
NODE_ID="node1"       # change per node
NODE_IP="192.168.1.10" # change per node

sudo tee /etc/openbao/config.hcl > /dev/null <<EOF
storage "raft" {
  path    = "/opt/openbao/data"
  node_id = "${NODE_ID}"

  retry_join {
    leader_api_addr     = "https://192.168.1.10:8200"
    leader_ca_cert_file = "/opt/openbao/tls/openbao-cert.pem"
  }
  retry_join {
    leader_api_addr     = "https://192.168.1.11:8200"
    leader_ca_cert_file = "/opt/openbao/tls/openbao-cert.pem"
  }
  retry_join {
    leader_api_addr     = "https://192.168.1.12:8200"
    leader_ca_cert_file = "/opt/openbao/tls/openbao-cert.pem"
  }
}

listener "tcp" {
  address         = "0.0.0.0:8200"
  tls_cert_file   = "/opt/openbao/tls/openbao-cert.pem"
  tls_key_file    = "/opt/openbao/tls/openbao-key.pem"
  tls_min_version = "tls12"
}

seal "pkcs11" {
  lib         = "/usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so"
  token_label = "openbao-token"
  key_label   = "openbao-unseal-key"
  mechanism   = "0x1087"
}

api_addr     = "https://${NODE_IP}:8200"
cluster_addr = "https://${NODE_IP}:8201"

ui        = true
log_level = "info"
EOF

sudo chown openbao:openbao /etc/openbao/config.hcl
sudo chmod 0640 /etc/openbao/config.hcl
Enter fullscreen mode Exit fullscreen mode

Note: OpenBao ignores retry_join entries that point to itself.

2.2 Keepalived (VIP Failover)

The VIP (192.168.1.13) floats to whichever node is the current Raft leader.

Leader check script (same on all nodes):

sudo tee /etc/keepalived/check_bao_leader.sh > /dev/null <<'EOF'
#!/bin/bash
LEADER_STATUS=$(curl -ks https://localhost:8200/v1/sys/leader | jq -r '.is_self')
if [ "$LEADER_STATUS" = "true" ]; then
    exit 0
else
    exit 1
fi
EOF
sudo chmod +x /etc/keepalived/check_bao_leader.sh
Enter fullscreen mode Exit fullscreen mode

Keepalived config — run on each node, replacing NODE_IP, PEER1_IP, PEER2_IP:

Node NODE_IP PEER1_IP PEER2_IP
1 192.168.1.10 192.168.1.11 192.168.1.12
2 192.168.1.11 192.168.1.10 192.168.1.12
3 192.168.1.12 192.168.1.10 192.168.1.11 
NODE_IP="192.168.1.10"   # this node
PEER1_IP="192.168.1.11"  # other two
PEER2_IP="192.168.1.12"

sudo tee /etc/keepalived/keepalived.conf > /dev/null <<EOF
global_defs {
    enable_script_security
    script_user root
}

vrrp_script check_bao_leader {
    script "/etc/keepalived/check_bao_leader.sh"
    interval 2
    weight 50
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1

    unicast_src_ip ${NODE_IP}
    unicast_peer {
        ${PEER1_IP}
        ${PEER2_IP}
    }

    virtual_ipaddress {
        192.168.1.13/24
    }

    track_script {
        check_bao_leader
    }
}
EOF

sudo systemctl enable --now keepalived
Enter fullscreen mode Exit fullscreen mode

Adjust interface eth0 and the /24 CIDR to match your network.

Phase 3 — Node 1 Only: SoftHSM Token and Key

3.1 Generate Pins 

USERPIN=$(openssl rand -base64 32)
SOPIN=$(openssl rand -base64 32)

echo "USERPIN: $USERPIN"
echo "SOPIN:   $SOPIN"
Enter fullscreen mode Exit fullscreen mode

Save both immediately:

  • USERPIN — needed on all nodes during Phases 3-4. After vTPM sealing, destroy all copies.
  • SOPIN — store offline with recovery keys. Never on any node. Only used to reset a userpin if a vTPM is lost.

3.2 Initialize Token and Create AES-256 Key 

# Init token
sudo -u openbao SOFTHSM2_CONF=/etc/openbao/softhsm2.conf \
  softhsm2-util --init-token --free \
  --label "openbao-token" \
  --pin "$USERPIN" \
  --so-pin "$SOPIN"

# Create AES-256 unseal key
sudo -u openbao SOFTHSM2_CONF=/etc/openbao/softhsm2.conf \
  pkcs11-tool \
  --module /usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so \
  --token-label "openbao-token" \
  --login --pin "$USERPIN" \
  --keygen --key-type aes:32 \
  --label "openbao-unseal-key" --id 01

# Verify
sudo -u openbao SOFTHSM2_CONF=/etc/openbao/softhsm2.conf \
  pkcs11-tool \
  --module /usr/lib/x86_64-linux-gnu/softhsm/libsofthsm2.so \
  --token-label "openbao-token" \
  --login --pin "$USERPIN" \
  --list-objects
Enter fullscreen mode Exit fullscreen mode

Expected output includes:

Secret Key Object; AES length 32
  label:      openbao-unseal-key
  ID:         01
Enter fullscreen mode Exit fullscreen mode

3.3 Seal Pin to Node 1's vTPM 

sudo -u openbao tpm2_createprimary -C o -G rsa2048 -c /tmp/primary.ctx

printf '%s' "$USERPIN" | sudo -u openbao tpm2_create \
  -C /tmp/primary.ctx -i - \
  -u /etc/openbao/tpm2/pin.pub \
  -r /etc/openbao/tpm2/pin.priv

sudo rm /tmp/primary.ctx
sudo chmod 0400 /etc/openbao/tpm2/pin.pub /etc/openbao/tpm2/pin.priv
sudo chown openbao:openbao /etc/openbao/tpm2/pin.pub /etc/openbao/tpm2/pin.priv
Enter fullscreen mode Exit fullscreen mode

Verify:

sudo -u openbao tpm2_createprimary -C o -G rsa2048 -c /tmp/primary.ctx
sudo -u openbao tpm2_load \
  -C /tmp/primary.ctx \
  -u /etc/openbao/tpm2/pin.pub \
  -r /etc/openbao/tpm2/pin.priv \
  -c /tmp/pin.ctx
sudo -u openbao tpm2_unseal -c /tmp/pin.ctx
# Must match USERPIN
sudo rm /tmp/primary.ctx /tmp/pin.ctx
Enter fullscreen mode Exit fullscreen mode

3.4 Replicate SoftHSM Token to Nodes 2 and 3

Nodes 2 and 3 must have completed Phase 1 before this step.

sudo tar -czf /tmp/softhsm2-tokens.tar.gz -C /etc/openbao/softhsm2 tokens/

scp /tmp/softhsm2-tokens.tar.gz <user>@192.168.1.11:/tmp/
scp /tmp/softhsm2-tokens.tar.gz <user>@192.168.1.12:/tmp/

rm /tmp/softhsm2-tokens.tar.gz
Enter fullscreen mode Exit fullscreen mode

Phase 4 — Nodes 2 and 3: Import Token and Seal Pin

Run on each of Node 2 and Node 3.

4.1 Import SoftHSM Token 

sudo tar -xzf /tmp/softhsm2-tokens.tar.gz -C /etc/openbao/softhsm2/
sudo chown -R openbao:openbao /etc/openbao/softhsm2/tokens/
sudo chmod -R 700 /etc/openbao/softhsm2/tokens/
rm /tmp/softhsm2-tokens.tar.gz

# Verify
sudo -u openbao SOFTHSM2_CONF=/etc/openbao/softhsm2.conf \
  softhsm2-util --show-slots
# Must show slot with label "openbao-token"
Enter fullscreen mode Exit fullscreen mode

4.2 Seal Pin to This Node's vTPM 

read -rsp "Enter USERPIN: " USERPIN && echo

sudo -u openbao tpm2_createprimary -C o -G rsa2048 -c /tmp/primary.ctx

printf '%s' "$USERPIN" | sudo -u openbao tpm2_create \
  -C /tmp/primary.ctx -i - \
  -u /etc/openbao/tpm2/pin.pub \
  -r /etc/openbao/tpm2/pin.priv

sudo rm /tmp/primary.ctx
sudo chmod 0400 /etc/openbao/tpm2/pin.pub /etc/openbao/tpm2/pin.priv
sudo chown openbao:openbao /etc/openbao/tpm2/pin.pub /etc/openbao/tpm2/pin.priv

# Verify
sudo -u openbao tpm2_createprimary -C o -G rsa2048 -c /tmp/primary.ctx
sudo -u openbao tpm2_load \
  -C /tmp/primary.ctx \
  -u /etc/openbao/tpm2/pin.pub \
  -r /etc/openbao/tpm2/pin.priv \
  -c /tmp/pin.ctx
sudo -u openbao tpm2_unseal -c /tmp/pin.ctx
# Must match USERPIN
sudo rm /tmp/primary.ctx /tmp/pin.ctx

unset USERPIN
Enter fullscreen mode Exit fullscreen mode

Phase 5 — All 3 Nodes: Start Cluster

5.1 Start OpenBao

Run on all three nodes:

sudo systemctl start openbao
sudo systemctl status openbao
sudo journalctl -u openbao -f
Enter fullscreen mode Exit fullscreen mode

All nodes should start and reach a sealed state, waiting for initialization.

5.2 Initialize (Node 1 only, run once) 

export BAO_ADDR="https://192.168.1.10:8200"
export BAO_CACERT="/opt/openbao/tls/openbao-cert.pem"

bao operator init -recovery-shares=5 -recovery-threshold=3
Enter fullscreen mode Exit fullscreen mode

Save the output immediately:

  • 5 Recovery Keys — emergency recovery only, not for daily unsealing.
  • 1 Root Token — initial setup only.

After init, all nodes should auto-unseal via the SoftHSM/vTPM chain.

5.3 Verify 

export BAO_ADDR="https://192.168.1.10:8200"
export BAO_CACERT="/opt/openbao/tls/openbao-cert.pem"

bao status
bao login <root-token>
bao operator raft list-peers
Enter fullscreen mode Exit fullscreen mode

Expected:

Node ID   Address              State       Voter
node1     192.168.1.10:8201    leader      true
node2     192.168.1.11:8201    follower    true
node3     192.168.1.12:8201    follower    true
Enter fullscreen mode Exit fullscreen mode

5.4 Verify VIP 

# The VIP should resolve to the leader
curl -ks https://192.168.1.13:8200/v1/sys/leader | jq .
Enter fullscreen mode Exit fullscreen mode

5.5 Revoke Root Token 

bao token revoke <root-token>
Enter fullscreen mode Exit fullscreen mode

Do this immediately after initial setup. Re-generate via recovery keys when needed.

Phase 6 — Verify Auto-Unseal 

# On any node
sudo systemctl restart openbao

# Check from another node
export BAO_ADDR="https://192.168.1.11:8200"
export BAO_CACERT="/opt/openbao/tls/openbao-cert.pem"
bao status
# Sealed: false → auto-unseal worked
Enter fullscreen mode Exit fullscreen mode

Troubleshooting 

# Service logs
sudo journalctl -u openbao -f

# Manual pin unseal test
sudo -u openbao tpm2_createprimary -C o -G rsa2048 -c /tmp/primary.ctx
sudo -u openbao tpm2_load \
  -C /tmp/primary.ctx \
  -u /etc/openbao/tpm2/pin.pub \
  -r /etc/openbao/tpm2/pin.priv \
  -c /tmp/pin.ctx
sudo -u openbao tpm2_unseal -c /tmp/pin.ctx
sudo rm /tmp/primary.ctx /tmp/pin.ctx

# SoftHSM slots
sudo -u openbao SOFTHSM2_CONF=/etc/openbao/softhsm2.conf softhsm2-util --show-slots

# TPM status
systemctl status tpm2-abrmd
ls -la /dev/tpm0 /dev/tpmrm0
groups openbao   # must include tss

# Keepalived / VIP
systemctl status keepalived
ip addr show eth0   # look for 192.168.1.13
Enter fullscreen mode Exit fullscreen mode

If tpm2_createprimary fails with DBUS/TCTI errors inside the service, add Environment="TPM2TOOLS_TCTI=device:/dev/tpmrm0" to the [Service] section to bypass tpm2-abrmd and hit the TPM device directly. The openbao user needs r/w on /dev/tpmrm0 (granted via the tss group).

Production Notes

Topic Detail
Recovery keys Store offline in a physical safe. Break-glass only.
Root token Revoke after setup. Re-generate via recovery keys when needed.
SOPIN Store offline with recovery keys. Used to reset a userpin if a vTPM is lost.
USERPIN After all vTPMs are sealed, this value should exist nowhere except inside the three vTPMs. Destroy all copies.
vTPM state Pin is bound to each node's vTPM. VM migration without vTPM preservation = node can't auto-unseal. Include vTPM state in backups.
SoftHSM backup The AES-256 key lives in /etc/openbao/softhsm2/tokens/. Back up encrypted, offline. Loss of all three copies = unrecoverable encrypted raft data.
Raft quorum 2 of 3 nodes must be healthy. If 2 go down, the cluster loses quorum and becomes unavailable (no reads or writes).
Pin rotation Per-node: softhsm2-util --init-pin --token-label "openbao-token" --so-pin <sopin> --pin <new-pin>, then re-seal new pin to that node's vTPM, restart OpenBao.
vTPM recovery Use SOPIN to set a new userpin, seal to the replacement vTPM, restart. Node rejoins via raft retry_join.
Mechanism 0x1087 CKM_AES_GCM — AEAD AES required by OpenBao. Preferred over RSA OAEP for performance.
TLS cert rotation Replace cert/key on all nodes and restart. No seal/unseal impact. Ensure new cert covers all node IPs + VIP in SANs.

Top comments (0)