Event: HashiTalks: Africa 2025
Category: DevOps, HashiCorp Vault, Automation
Modern infrastructure depends on secure secret management, yet achieving high availability (HA) often feels like a luxury reserved for enterprise budgets.
In this blog, we’ll explore how you can build a highly available, production-grade HashiCorp Vault cluster using Raft Integrated Storage and the community edition of MinIO (an S3-compatible object storage) for resilient backups. Everything was implemented in my homelab.
Why This Approach?
Many organizations, especially those operating on tight budgets or in hybrid environments, require strong security but cannot always afford enterprise secret management service.
Instead of depending on Vault Enterprise with external storage solutions such as Consul, etcd, or DynamoDB, my approach leverages:
- Vault OSS (Open Source) for secret management
- Raft storage for native leader election and data replication
- MinIO for S3-compatible snapshot backups
This setup provides HA, fault tolerance, and easy disaster recovery, all without extra licensing costs.
Architecture Overview
Here’s what the setup looks like:
Each Vault node runs in HA mode using Raft. The leader handles writing and replicating data to followers automatically.
Snapshots are taken periodically and pushed to MinIO for offsite/off-cluster recovery.
Note: All three servers used in this setup are provisioned within a homelab environment.
Step 1: Prepare Your Environment
Provision 3 servers (VMs or bare metal).
Ensure network connectivity between them and install:
sudo apt install vault
sudo mkdir -p /opt/vault/tls /opt/vault/data /opt/vault/backups
Create a Local Certificate Authority (CA)
Vault requires TLS for secure communication. I’ll start by generating my own self-signed CA and use it to issue Vault’s server certificate.
Generate a Private Key for the CA
openssl genrsa -out vault-ca-key.pem 4096
Create a Self-Signed CA Certificate
openssl req -x509 -new -nodes -key vault-ca-key.pem \
-subj "/CN=Vault-CA" \
-days 3650 -out vault-ca-cert.pem
Generate a Vault Server Private Key and CSR; This should be done for all the vault servers.
openssl genrsa -out vault-server-key.pem 2048
openssl req -new -key vault-server1.key -out vault-server-1.csr -config vault-server1.conf
Below is the content of vault-server1.conf; update accordingly for other vault instances.
~/vault-cert/vault-ca (0.146s)
cat vault-server1.conf
[req]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = v3_req
distinguished_name = dn
[dn]
C = NG
ST = Abuja
L = Abuja
O = VaultOrg
OU = DevOps
CN = vault-server-1
[v3_req]
subjectAltName = @alt_names
[alt_names]
DNS.1 = vault-server-1
DNS.2 = vault1.internal.local
DNS.3 = 192-168-64-5.nip.io
IP.1 = 192.168.64.5
Sign the Server Certificate with the CA
openssl x509 -req -in vault-server-1.csr \
-CA vault-ca-cert.pem -CAkey vault-ca-key.pem -CAcreateserial \
-out vault-server-cert1.pem -days 365 -sha256
Copy Certificates
Move your generated certs to /opt/vault/tls:
sudo mkdir -p /opt/vault/tls
sudo cp vault-server-cert1.pem vault-server-key.pem vault-ca-cert.pem /opt/vault/tls
Step 2: Configure Vault for HA Using Raft Storage
Configure accordingly on all three vault servers.
make the certificate name on each server uniform as shown in the vault.hcl file below.
Edit /etc/vault.d/vault.hcl:
root@vault-server-1:/opt/vault/backups# cat /etc/vault.d/vault.hcl
1 ui = true
2 disable_mlock = true
3
4 storage "raft" {
5 path = "/opt/vault/data"
6 node_id = "vault-server-1"
7
8 retry_join {
9 leader_tls_servername = "vault-server-1"
10 leader_api_addr = "https://192.168.64.5:8200"
11 leader_ca_cert_file = "/opt/vault/tls/vault-ca.pem"
12 leader_client_cert_file = "/opt/vault/tls/vault-cert.pem"
13 leader_client_key_file = "/opt/vault/tls/vault-key.pem"
14 }
15
16 retry_join {
17 leader_tls_servername = "vault-server-2"
18 leader_api_addr = "https://192.168.64.6:8200"
19 leader_ca_cert_file = "/opt/vault/tls/vault-ca.pem"
20 leader_client_cert_file = "/opt/vault/tls/vault-cert.pem"
21 leader_client_key_file = "/opt/vault/tls/vault-key.pem"
22 }
23
24 retry_join {
25 leader_tls_servername = "vault-server-3"
26 leader_api_addr = "https://192.168.64.7:8200"
27 leader_ca_cert_file = "/opt/vault/tls/vault-ca.pem"
28 leader_client_cert_file = "/opt/vault/tls/vault-cert.pem"
29 leader_client_key_file = "/opt/vault/tls/vault-key.pem"
30 }
31 }
32
33 listener "tcp" {
34 address = "0.0.0.0:8200"
35 tls_cert_file = "/opt/vault/tls/vault-cert.pem"
36 tls_key_file = "/opt/vault/tls/vault-key.pem"
37 }
38
39 api_addr = "https://192.168.64.5:8200"
40 cluster_addr = "https://192.168.64.5:8201"
Start and enable Vault:
systemctl enable vault
systemctl start vault
root@vault-server-1:/opt/vault/tls# systemctl status vault
● vault.service - "HashiCorp Vault - A tool for managing secrets"
Loaded: loaded (/usr/lib/systemd/system/vault.service; enabled; preset: enabled)
Active: active (running) since Sun 2025-09-28 18:53:51 UTC; 23s ago
Main PID: 8944 (vault)
Tasks: 8 (limit: 4549)
Memory: 23.9M (peak: 24.1M)
CPU: 750ms
CGroup: /system.slice/vault.service
└8944 /usr/bin/vault server -config=/etc/vault.d/vault.hcl
Initialize and unseal Vault, then join other nodes to form a Raft cluster.
Initialize Vault on one of the vault nodes
$ vault operator init -key-shares=5 -key-threshold=3
Get "https://192.168.64.5:8200/v1/sys/seal-status": tls: failed to verify certificate: x509: certificate signed by unknown authority
$ export VAULT_ADDR="https://192.168.64.5:8200"
$ export VAULT_CACERT="/opt/vault/tls/vault-ca.pem"
$ echo 'export VAULT_CACERT="/opt/vault/tls/vault-ca.pem"' > ~/.bashrc
$ source ~/.bashrc
$ vault operator init -key-shares=5 -key-threshold=3
Unseal Key 1: oh0oYmK6oQkdU9oPQF35nMYKgZoZEhFHS40adcTK13Xh
Unseal Key 2: I7gby7hcwW/njhGiPVLOVBTBgUQGk50C7+FQZKtLKUvq
Unseal Key 3: OzeDrBJJ@epEonEFc69Nctj5uvhW+u9+tVMEo4jQSkeP
Unseal Key 4: ryf5ZXCmaulkseIYmOXIRAavOWEOvDdSbu7sdinUvnfU
Unseal Key 5: 87BDV1ithJhk8Abtb5SIDpeXtdH3/p70nwc650S1eUm+
Initial Root Token: hvs.zK4B84b2f5pyCgeYtv5WG8RR
Vault initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above. When the Vault is re-sealed, restarted, or stopped, you must supply at least 3 of these keys to unseal it before it can start servicing requests.
Vault does not store the generated root key. Without at least 3 keys to reconstruct the root key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of existing unseal keys shares. See "vault operator rekey" for more information.
root@vault-server-1:/opt/vault/tls# vault operator unseal
Unseal Key (will be hidden):
Key Value
--- ----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 2/3
Unseal Nonce d4ab9730-3088-e2b7-33bc-b4e69ec1b568
Version 1.20.4
Build Date 2025-09-23T13:22:38Z
Storage Type raft
Removed From Cluster false
HA Enabled true
Run vault status command to view the current state of the cluster
root@vault-server-1:/opt/vault/tls# vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.20.4
Build Date 2025-09-23T13:22:38Z
Storage Type raft
Cluster Name vault-cluster-edcc5476
Cluster ID 130c35ea-023d-c34e-4d69-c217fa974d04
Removed From Cluster false
HA Enabled true
HA Cluster https://192.168.64.5:8201
HA Mode active
Active Since 2025-09-28T19:11:36.208910265Z
Raft Committed Index 37
Raft Applied Index 37
Verify Cluster
Check Raft peers:
vault operator raft list-peers
Get current leader: vault operator raft list-peers
root@vault-server-1:/opt/vault/tls# vault operator raft list-peers
Node Address State Voter
vault-server-1 192.168.64.5:8201 leader true
vault-server-2 192.168.64.6:8201 follower true
vault-server-3 192.168.64.7:8201 follower true
root@vault-server-1:/opt/vault/tls#
Open the UI page of the vault and log in with root token (this token was generated alongside sealed keys during initialization) to further confirm status.
Step 3: Enable KV Secrets Engine (Version 2)
Run the below command on the leader node.
vault secrets enable -path=secret kv-v2
Create a key-value pair:
vault kv put secret/demo event="HashiTalksAfrica" year="2025" location="Online"
Verify:
vault kv get secret/demo
===== Secret Path =====
secret/data/demo
======= Metadata =======
Key Value
--- -----
created_time 2025-10-04T07:42:19.123456Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
========== Data ==========
Key Value
--- -----
event HashiTalksAfrica
year 2025
location Online
Step 4: Automate Vault Snapshot Backups to MinIO
Follow the link to install MinIO on a server (as a container or system service)or use AWS S3 if you prefer it.
root@vault-server-2:~# ssh root@192.168.64.4
The authenticity of host '192.168.64.4 (192.168.64.4)' can't be established.
ECDSA key fingerprint is SHA256:Ld5Gu7HBMnLLTYhVQzFvuYfPODcUwEZOHLBKXXXXXX.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.64.4' (ECDSA) to the list of known hosts.
root@192.168.64.4's password:
root@minio-server:~# mkdir /opt/minio
root@minio-server:~# cd /opt/minio
# Download the latest MinIO binary for Linux/amd64
root@minio-server:/opt/minio# wget https://dl.min.io/server/minio/release/linux-amd64/minio
--2025-10-08 06:46:00-- https://dl.min.io/server/minio/release/linux-amd64/minio
Resolving dl.min.io (dl.min.io)... 178.128.69.202, 138.68.11.125
Connecting to dl.min.io (dl.min.io)|178.128.69.202|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 74123264 (71M) [application/octet-stream]
Saving to: 'minio'
minio 100%[=======================] 70.71M 4.14MB/s in 17s
2025-10-08 06:46:17 (4.14 MB/s) - 'minio' saved [74123264/74123264]
# Make the binary executable
root@minio-server:/opt/minio# chmod +x minio
# Create a systemd service file for MinIO
root@minio-server:/opt/minio# cat > /etc/systemd/system/minio.service <<EOF
[Unit]
Description=MinIO
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/opt/minio/minio
[Service]
WorkingDirectory=/opt/minio/
ExecStart=/opt/minio/minio server /opt/minio/data
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
# Start and enable the MinIO service
root@minio-server:/opt/minio# systemctl enable minio
Created symlink /etc/systemd/system/multi-user.target.wants/minio.service → /etc/systemd/system/minio.service.
root@minio-server:/opt/minio# systemctl start minio
Install MinIO client (mc) on all vault servers; This client will enable the vault server to run mc command against the minIO server.
# Download the correct ARM64 version
wget https://dl.min.io/client/mc/release/linux-arm64/mc -O /usr/local/bin/mc
# Make it executable
chmod +x /usr/local/bin/mc
# Verify installation
mc --version
-- 2025-10-05 06:40:05 -- https://dl.min.io/client/mc/release/linux-arm64/mc
Resolving dl.min.io (dl.min.io)... 178.128.69.202, 138.68.11.125
Connecting to dl.min.io (dl.min.io)|178.128.69.202|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 28704952 (27M) [application/octet-stream]
Saving to: '/usr/local/bin/mc'
/usr/local/bin/mc 100%[=======================] 27.37M 2.71MB/s in 10s
2025-10-05 06:40:16 (2.71 MB/s) - '/usr/local/bin/mc' saved [28704952/28704952]
mc version RELEASE.2025-08-13T08-35-41Z (commit-id=7394ce0dd2a80935aded936b09fa12cbb3cb8096)
Runtime: go1.24.6 linux/arm64
Copyright (c) 2015-2025 MinIO, Inc.
License GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
root@vault-server-2:~#
Add MinIO endpoint alias on all vault servers
Create a minio bucket with the name vault using the below command:
PS: You can run the command on the vault node or on the MinIO server, you can also create the bucket from the UI.
# Create the bucket
root@vault-server-1:~# mc mb myminio/vault
root@vault-server-1:~# mc mb myminio/vault
[2025-28-08 20:55:07 UTC] 0B vault
Step 5: Verify Cluster Backup
Firstly, create a backup manually and push to minio before automating the process with a shell script vault-backup-to-minio.sh, systemd service and a systemd timer to trigger the service at predefined intervals.
Automating Backup
Backup Script — /usr/local/bin/vault-backup-to-minio.sh
PS: configure the below steps on all the vault instances.
#!/usr/bin/env bash
set -euo pipefail
#########################CONFIGURATION########################
VAULT_ADDR="${VAULT_ADDR:-https://192.168.64.5:8200}"
VAULT_CACERT="${VAULT_CACERT:-/opt/vault/tls/vault-ca.pem}"
BACKUP_DIR="${BACKUP_DIR:-/opt/vault/backups}"
MINIO_ALIAS="${MINIO_ALIAS:-myminio}"
MINIO_BUCKET="${MINIO_BUCKET:-vault}"
RETENTION_DAYS="${RETENTION_DAYS:-14}"
LOGFILE="${LOGFILE:-/var/log/vault-backup.log}"
MINIO_ACCESS_KEY=${MINIO_ACCESS_KEY:-$(cat /root/.accesskey)}
MINIO_SECRET_KEY=${MINIO_SECRET_KEY:-$(cat /root/.secretkey)}
###############################################################
export MC_CONFIG_DIR=/opt/vault/mc-config
mkdir -p $MC_CONFIG_DIR
mc alias set myminio http://192.168.64.4:9000 ${MINIO_ACCESS_KEY} ${MINIO_SECRET_KEY}
mkdir -p "$BACKUP_DIR"
touch "$LOGFILE"
# restrict log file
chmod 600 "$LOGFILE"
echo "$(date -u '+%F %T') INFO: starting vault backup" > "$LOGFILE"
# 1) Check that this node is active (leader).
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" --cacert "$VAULT_CACERT" "${VAULT_ADDR}/v1/sys/health?standbyok=false" || echo "000")
if [ "$HTTP_CODE" -eq 200 ]; then
echo "$(date) INFO: this node is active (HTTP 200)" >> "$LOGFILE"
elif [ "$HTTP_CODE" -eq 429 ]; then
echo "$(date) INFO: this node is standby (HTTP 429) - exiting" >> "$LOGFILE"
exit 1
elif [ "$HTTP_CODE" -eq 503 ]; then
echo "$(date) ERROR: vault sealed or not initialized (HTTP 503) - exiting" >> "$LOGFILE"
exit 1
else
echo "$(date) ERROR: unexpected vault health HTTP code: $HTTP_CODE - exiting" >> "$LOGFILE"
exit 1
fi
# 2) Create snapshot
TIMESTAMP=$(date +'%F-%H%M%S')
SNAPFILE="${BACKUP_DIR}/vault-${TIMESTAMP}.snap"
echo "$(date) INFO: creating snapshot $SNAPFILE" >> "$LOGFILE"
if vault operator raft snapshot save "$SNAPFILE"; then
echo "$(date) INFO: snapshot created: $SNAPFILE" >> "$LOGFILE"
else
echo "$(date) ERROR: snapshot creation failed" >> "$LOGFILE"
exit 2
fi
# 3) Compress (saves bandwidth)
if gzip -f "$SNAPFILE"; then
SNAPFILE="${SNAPFILE}.gz"
echo "$(date) INFO: compressed snapshot to $SNAPFILE" >> "$LOGFILE"
else
echo "$(date) WARN: gzip failed, continuing with uncompressed snap" >> "$LOGFILE"
fi
# 4) Upload to MinIO
echo "$(date) INFO: uploading $SNAPFILE to ${MINIO_ALIAS}/${MINIO_BUCKET}/" >> "$LOGFILE"
if mc cp "$SNAPFILE" "${MINIO_ALIAS}/${MINIO_BUCKET}/"; then
echo "$(date) INFO: upload successful" >> "$LOGFILE"
else
echo "$(date) ERROR: upload to MinIO failed" >> "$LOGFILE"
exit 3
fi
# 5) Local retention - delete local files older than RETENTION_DAYS
echo "$(date) INFO: removing local snapshots older than ${RETENTION_DAYS} days" >> "$LOGFILE"
find "$BACKUP_DIR" -type f -name 'vault-*.snap*' -mtime +"$RETENTION_DAYS" -print -delete >> "$LOGFILE" 2>&1 || true
echo "$(date) INFO: vault backup completed" >> "$LOGFILE"
exit 0
Make it executable:
chmod +x /usr/local/bin/vault-backup-to-minio.sh
Systemd Unit: /etc/systemd/system/vault-backup.service
[Unit]
Description=Vault Raft Snapshot Backup to MinIO
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/vault-backup-to-minio.sh
User=root
Group=root
# Protect system a bit more
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
Timer: /etc/systemd/system/vault-backup.timer
[Unit]
Description=Run Vault backup script every 6 hours
[Timer]
OnCalendar=*:0/6 # every 6 hours on the hour
Persistent=true # if missed while down, run at boot
[Install]
WantedBy=timers.target
Enable and start:
systemctl daemon-reload
systemctl enable --now vault-backup.timer
Step 6: Delete Secrets and Restore from MinIO
Delete the secret:
vault kv delete -versions=1 secret/demo
vault kv metadata delete secret/demo
Download snapshot:
mc cp myminio/vault-backups/vault-snapshots/vault-2025-10-06-064537.snap.gz /opt/vault/backups/
Unzip and restore:
gunzip vault-2025-10-06-064537.snap.gz
vault operator raft snapshot restore vault-2025-10-06-064537.snap
Verify the secret is restored:
vault kv get secret/demo
===== Secret Path =====
secret/data/demo
======= Metadata =======
Key Value
--- -----
created_time 2025-10-04T07:42:19.123456Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
========== Data ==========
Key Value
--- -----
event HashiTalksAfrica
year 2025
location Online
Conclusion
By combining Vault’s integrated storage (Raft) with MinIO, you will be able to build a highly available and resilient Vault setup—entirely with open-source tools and minimal cost.
This approach ensures that even if all Vault nodes fail, you can restore your data from a secure snapshot stored in MinIO, getting back to operational state in minutes. It’s a practical solution for teams running on-premises or in budget-conscious environments, where traditional enterprise Vault may not be feasible.
However, while this setup provides HA and data durability, there’s one more layer to make it truly production-ready:
Recommendation
Place a load balancer (such as Nginx, HAProxy, or an L4 balancer like Keepalived) in front of your Vault cluster.
The load balancer should:
Continuously check the health and leadership status of Vault nodes using the /v1/sys/health or /v1/sys/leader endpoint.
Forward client traffic only to the current leader node, since Vault accepts writes exclusively on the leader.
Automatically re-route requests to the new leader after failover, ensuring seamless continuity for applications and users.
This way, your Vault deployment achieves:
- High Availability
- Data Durability
- Automatic Failover
- Seamless Client Experience
With this foundation, you now have a production-grade, open-source Vault cluster that’s cost-effective, recoverable, and resilient against both node and data failures.
Top comments (1)
This is beautify and insightful. Thank you.