DEV Community

Cover image for Building a High-Availability Vault Cluster with Docker and Raft Storage
Kubilay İşen
Kubilay İşen

Posted on

Building a High-Availability Vault Cluster with Docker and Raft Storage

Vault Cluster Architecture

Introduction

HashiCorp Vault is one of the most powerful secrets management solutions in the industry. However, setting up a production-ready, highly-available Vault cluster can be intimidating. In this article, I'll walk you through building a 3-node Vault cluster using Docker with automatic unsealing, Raft-based storage, and infrastructure-as-code automation.

By the end of this guide, you'll have a resilient secrets management infrastructure that can handle node failures and scale horizontally.


Why Vault? Why High-Availability?

The Problem

In modern infrastructure:

  • Secrets are scattered across multiple systems (databases, APIs, certificates)
  • No single source of truth for credential rotation
  • Compliance requirements demand audit trails
  • Manual secret management is error-prone

The Solution

Vault provides:

  • Centralized secret management - Single source of truth
  • Encryption as a service - Encrypt/decrypt without exposing keys
  • Dynamic credentials - Automatically generate short-lived credentials
  • Audit logging - Complete trail of who accessed what and when
  • High availability - Never lose access to your secrets

Architecture Overview

┌─────────────────────────────────────────────────────────┐
│                 Docker Compose Network                  │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐   │
│  │  Vault Node  │  │  Vault Node  │  │  Vault Node  │   │
│  │     (1)      │  │     (2)      │  │     (3)      │   │
│  └──────────────┘  └──────────────┘  └──────────────┘   │
│         │                 │                  │          │
│  ┌──────┴─────────────────┼──────────────────┴────┐     │
│  │                  Raft Consensus                │     │
│  └────────────────────────────────────────────────┘     │
│                                                         │
│  ┌──────────────────────────────────────────────────┐   │
│  │   Nginx Load Balancer (Optional)                 │   │
│  └──────────────────────────────────────────────────┘   │
│                                                         │
└─────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Our setup features:

  • 3 Vault nodes for true high-availability
  • Raft storage backend - No external dependencies (unlike Consul)
  • Auto-unsealing - Automatic node recovery without manual intervention
  • Docker Compose orchestration - Easy to deploy and manage
  • Health checks - Automatic failure detection

Prerequisites

Before we begin, make sure you have:

- Docker & Docker Compose (v3.8+)
- Taskfile CLI for [go-task](https://taskfile.dev/#/installation) automation
- curl (for API testing)
Enter fullscreen mode Exit fullscreen mode

Install the requirements:

# macOS
brew install docker docker-compose task jq curl

# Linux
sudo apt-get install docker.io docker-compose taskfile jq curl
Enter fullscreen mode Exit fullscreen mode

Project Structure

vault-docker-cluster/
├── docker-compose.yaml           # Services definition
├── Dockerfile.vault              # Custom Vault image
├── Taskfile.yml                  # Task automation
├── init-and-generate-unseal.sh   # Cluster initialization
├── auto-unseal-monitor.sh        # Monitoring script
├── vault-1/
│   ├── config/
│   │   ├── vault.hcl            # Vault configuration
│   │   └── unseal.sh            # Auto-generated unseal script
│   └── data/                     # Raft data storage
├── vault-2/                      # (same structure)
└── vault-3/                      # (same structure)
Enter fullscreen mode Exit fullscreen mode

Step 1: Docker Compose Configuration

Let's start with the docker-compose.yaml. This file orchestrates three Vault nodes:

version: '3.8'

networks:
  vault_net:
    driver: bridge

services:
  vault-1:
    image: vaultdockercluster:1.20
    restart: unless-stopped
    volumes:
      - ./vault-1/config:/vault/config
      - ./vault-1/data:/vault/data
    cap_add:
      - IPC_LOCK                    # Required for mlock
    entrypoint:
      - vault
      - server
      - -config=/vault/config/vault.hcl
    ulimits:
      memlock: -1                   # Unlimited memory lock
      nofile:
        soft: 65535
        hard: 65535
    networks:
      - vault_net
    healthcheck:
      test: ["CMD", "vault", "status", "-tls-skip-verify"]
      interval: 10s
      timeout: 5s
      retries: 5
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 1G

  # vault-2 and vault-3 use the same configuration...
Enter fullscreen mode Exit fullscreen mode

Key Configuration Points:

Setting Purpose
IPC_LOCK Allows Vault to lock memory pages (prevents swapping secrets to disk)
memlock: -1 Unlimited memory lock for all processes
healthcheck Detects node failures automatically
ulimits Handles many concurrent connections
networks Isolated network for inter-node communication

Step 2: Vault Configuration (HCL)

Each node has its own vault.hcl configuration:

vault-1:

storage "raft" {
  path    = "/vault/data"
  node_id = "vault-1"
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_disable   = true
}

api_addr     = "http://vault-1:8200"
cluster_addr = "http://vault-1:8201"

ui = true
disable_mlock = true
Enter fullscreen mode Exit fullscreen mode

vault-2:

storage "raft" {
  path    = "/vault/data"
  node_id = "vault-2"

  retry_join {
    leader_api_addr = "http://vault-1:8200"
  }
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_disable   = true
}

api_addr     = "http://vault-2:8200"
cluster_addr = "http://vault-2:8201"

ui = true
disable_mlock = true
Enter fullscreen mode Exit fullscreen mode

vault-3:

storage "raft" {
  path    = "/vault/data"
  node_id = "vault-3"

  retry_join {
    leader_api_addr = "http://vault-1:8200"
  }
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_disable   = true
}

api_addr     = "http://vault-3:8200"
cluster_addr = "http://vault-3:8201"

ui = true
disable_mlock = true
Enter fullscreen mode Exit fullscreen mode

Raft vs Other Backends:

Aspect Raft Consul S3
Setup Complexity Simple Complex Simple
Dependencies None Consul cluster needed AWS account
Cost Free Free (self-hosted) $ per API call
Performance Excellent Good Slower
Best For Self-hosted HA Enterprise AWS-native

Step 3: Automated Setup with Taskfile

The Taskfile.yml automates common operations:

version: '3'

vars:
  VAULT_ADDR: 'http://127.0.0.1:8200'
  KEY_SHARES: 5
  KEY_THRESHOLD: 3

tasks:
  bootstrap:
    desc: Full bootstrap - start cluster, initialize, and setup
    cmds:
      - task: up
      - echo "⏳ Waiting for containers to be ready (15 seconds)..."
      - sleep 15
      - task: init
      - echo ""
      - echo "📋 Credentials saved! Review them above."
      - echo "Press Enter to continue with cluster setup..."
      - read _
      - task: setup-cluster
      - echo ""
      - echo "✅ Bootstrap complete!"

  up:
      desc: Start the Vault cluster
      cmds:
        - docker-compose up -d

  init:
    desc: Initialize vault-1 and auto-generate unseal.sh scripts
    cmds:
      - ./init-and-generate-unseal.sh
    preconditions:
      - sh: docker-compose ps vault-1 | grep -q "Up"
        msg: "vault-1 is not running. Run 'task up' first."
      - sh: command -v jq >/dev/null 2>&1
        msg: "jq is required but not installed. Install with: brew install jq"

  setup-cluster:
    desc: Complete cluster setup - unseal vault-1, join and unseal vault-2 and vault-3
    cmds:
      - task: unseal-vault-1
      - task: join-vault-2
      - task: unseal-vault-2
      - task: join-vault-3
      - task: unseal-vault-3
Enter fullscreen mode Exit fullscreen mode

Usage is straightforward:

# Start everything
task bootstrap

# Or step by step
task up
task init
task setup-cluster
# Check status
docker-compose ps
Enter fullscreen mode Exit fullscreen mode

Step 4: Initialization and Unsealing

The initialization script (init-and-generate-unseal.sh) handles the critical setup:

  • Calls vault operator init against vault-1.
  • Prints the unseal keys + root token (and writes them to vault-credentials-<timestamp>.md and vault-init-keys.json).
  • Generates vault-*/config/unseal.sh helper scripts.
#!/bin/bash
set -e

echo "🔐 Initializing Vault Cluster..."
echo ""

# Wait for vault-1 to be ready (unsealed but not initialized)
echo "⏳ Waiting for vault-1 to be ready..."
max_attempts=30
attempt=0

while [ $attempt -lt $max_attempts ]; do
    # Check if we can connect to Vault and get a status response
    # vault status returns non-zero exit code when sealed, so we need to capture both stdout and exit code
    status=$(docker-compose exec -T vault-1 sh -c "export VAULT_ADDR='http://127.0.0.1:8200' && vault status -format=json 2>&1" || true)

    # Check if we got valid JSON output (meaning Vault is responding)
    if echo "$status" | jq -e . >/dev/null 2>&1; then
        initialized=$(echo "$status" | jq -r '.initialized // false')

        if [ "$initialized" = "false" ]; then
            echo "✅ Vault is ready for initialization"
            break
        elif [ "$initialized" = "true" ]; then
            echo "❌ Error: Vault is already initialized!"
            echo ""
            echo "If you want to re-initialize:"
            echo "  1. Run: task reset"
            echo "  2. Run: task bootstrap"
            exit 1
        fi
    else
        # Vault is not responding yet, keep waiting
        echo "⏳ Waiting for Vault to start (attempt $((attempt + 1))/$max_attempts)..."
    fi

    attempt=$((attempt + 1))
    sleep 2
done

if [ $attempt -eq $max_attempts ]; then
    echo "❌ Timeout waiting for vault-1 to be ready"
    echo ""
    echo "Check logs with: docker-compose logs vault-1"
    exit 1
fi

echo ""

# Initialize vault-1 and capture output
echo "🔑 Initializing Vault..."
INIT_OUTPUT=$(docker-compose exec -T vault-1 sh -c "export VAULT_ADDR='http://127.0.0.1:8200' && vault operator init -key-shares=5 -key-threshold=3 -format=json")

# Parse the JSON output
UNSEAL_KEY_1=$(echo "$INIT_OUTPUT" | jq -r '.unseal_keys_b64[0]')
UNSEAL_KEY_2=$(echo "$INIT_OUTPUT" | jq -r '.unseal_keys_b64[1]')
UNSEAL_KEY_3=$(echo "$INIT_OUTPUT" | jq -r '.unseal_keys_b64[2]')
UNSEAL_KEY_4=$(echo "$INIT_OUTPUT" | jq -r '.unseal_keys_b64[3]')
UNSEAL_KEY_5=$(echo "$INIT_OUTPUT" | jq -r '.unseal_keys_b64[4]')
ROOT_TOKEN=$(echo "$INIT_OUTPUT" | jq -r '.root_token')

echo "════════════════════════════════════════════════════════════════"
echo "⚠️  SAVE THESE CREDENTIALS SECURELY - THEY CANNOT BE RECOVERED!"
echo "════════════════════════════════════════════════════════════════"
echo ""
echo "Unseal Key 1: $UNSEAL_KEY_1"
echo "Unseal Key 2: $UNSEAL_KEY_2"
echo "Unseal Key 3: $UNSEAL_KEY_3"
echo "Unseal Key 4: $UNSEAL_KEY_4"
echo "Unseal Key 5: $UNSEAL_KEY_5"
echo ""
echo "Root Token: $ROOT_TOKEN"
echo ""
echo "════════════════════════════════════════════════════════════════"
echo ""

# Save to a backup file
BACKUP_FILE="vault-credentials-$(date +%Y%m%d-%H%M%S).md"
cat > "$BACKUP_FILE" << EOF
Vault Cluster Initialization - $(date)
════════════════════════════════════════════════════════════════

Unseal Key 1: $UNSEAL_KEY_1
Unseal Key 2: $UNSEAL_KEY_2
Unseal Key 3: $UNSEAL_KEY_3
Unseal Key 4: $UNSEAL_KEY_4
Unseal Key 5: $UNSEAL_KEY_5

Root Token: $ROOT_TOKEN

════════════════════════════════════════════════════════════════
⚠️  Store this file securely and delete it from this location!
EOF

echo "✅ Credentials saved to: $BACKUP_FILE"
echo ""

# Generate unseal.sh for vault-1
echo "📝 Generating unseal.sh scripts..."

cat > vault-1/config/unseal.sh << EOF
#!/bin/sh
set -e
export VAULT_ADDR='http://127.0.0.1:8200'
vault operator unseal $UNSEAL_KEY_1
vault operator unseal $UNSEAL_KEY_2
vault operator unseal $UNSEAL_KEY_3
echo "✅ Vault unsealed successfully"
EOF

chmod +x vault-1/config/unseal.sh

# Copy to vault-2
cp vault-1/config/unseal.sh vault-2/config/unseal.sh

# Copy to vault-3
cp vault-1/config/unseal.sh vault-3/config/unseal.sh

echo "✅ Created unseal.sh in vault-1/config/"
echo "✅ Created unseal.sh in vault-2/config/"
echo "✅ Created unseal.sh in vault-3/config/"
echo ""

# Also save JSON format for automation
echo "$INIT_OUTPUT" | jq '.' > vault-init-keys.json
echo "✅ Saved JSON format to: vault-init-keys.json"
echo ""

echo "════════════════════════════════════════════════════════════════"
echo "🎉 Initialization Complete!"
echo "════════════════════════════════════════════════════════════════"
echo ""
echo "Next steps:"
echo "  1. Secure the credentials file: $BACKUP_FILE"
echo "  2. Run: task setup-cluster"
echo "  3. Or manually:"
echo "     - task unseal-vault-1"
echo "     - task join-vault-2 && task unseal-vault-2"
echo "     - task join-vault-3 && task unseal-vault-3"
echo ""
Enter fullscreen mode Exit fullscreen mode

Important Security Notes:

⚠️ Critical: Store the root token and unseal keys safely:

  • Save them in a password manager
  • Never commit them to version control
  • Consider using Vault's auto-unseal with KMS (AWS, GCP, Azure)

Step 5: Building the Docker Image

A minimal Dockerfile.vault:

FROM hashicorp/vault:1.20
ENV TZ=Europe/Istanbul
Enter fullscreen mode Exit fullscreen mode

Build it with:

docker build -f Dockerfile.vault -t vaultdockercluster:1.20 .
Enter fullscreen mode Exit fullscreen mode

Step 6: Raft Cluster Formation

After initializing vault-1, join the other nodes:

# Join vault-2 to the cluster
docker-compose exec vault-2 sh -c \
  "vault operator raft join http://vault-1:8200"

# Join vault-3 to the cluster
docker-compose exec vault-3 sh -c \
  "vault operator raft join http://vault-1:8200"

# Verify cluster status
docker-compose exec vault-1 vault operator raft list-peers
Enter fullscreen mode Exit fullscreen mode

Expected output:

Node ID    Address            State       Voter
------     -------            -----       -----
vault-1    vault-1:8201       leader      true
vault-2    vault-2:8201       follower    true
vault-3    vault-3:8201       follower    true
Enter fullscreen mode Exit fullscreen mode

Complete Workflow: From Zero to Hero

Here's the complete startup sequence:

# 1. Start the containers
task up

# 2. Watch the logs
docker-compose logs -f vault-1

# 3. Initialize the cluster
task init

# Save the credentials somewhere safe!

# 4. Unseal all nodes
task unseal-all

# 5. Join nodes 2 and 3 to the cluster
task join-vault-2
task join-vault-3

# 6. Verify the cluster
docker-compose exec vault-1 vault operator raft list-peers

# 7. Access the UI
open http://localhost:8200/ui
Enter fullscreen mode Exit fullscreen mode

http://localhost:8200/ui


Testing Your Cluster

Test 1: High Availability

Kill the leader node and watch the cluster recover:

# Kill vault-1 (the leader)
docker-compose kill vault-1

# Check who's the new leader
docker-compose exec vault-2 vault operator raft list-peers

# Bring it back
docker-compose restart vault-1

# Verify recovery
docker-compose ps
Enter fullscreen mode Exit fullscreen mode

Auto-Unseal Monitor

When a Vault container restarts, the auto-unseal monitor automatically detects that the Vault node has become sealed and immediately unseals it using the stored unseal keys.

The auto-unseal-monitor.sh script runs in its own container (vault-unsealer).

  • Polls each Vault node every 30 seconds via vault status.
  • If a node transitions to sealed, reads the unseal keys from the mounted unseal.sh file and runs the three vault operator unseal commands.
  • Retries up to three times per incident. Logs show timestamps and outcomes.

Because the helper executes the same unseal script stored on disk, protect the vault-*/config/unseal.sh files and remove them when no longer needed.

Test 2: Secret Storage

Store and retrieve a secret:


docker-compose exec vault-1 sh

# Login
export VAULT_TOKEN="your-root-token"
export VAULT_ADDR="http://localhost:8200"

# Create a secret
vault kv put secret/my-app/database \
  username=admin \
  password=super-secret-password

# Retrieve it
vault kv get secret/my-app/database

# Read it via API
curl -H "X-Vault-Token: $VAULT_TOKEN" \
  http://localhost:8200/v1/secret/data/my-app/database
Enter fullscreen mode Exit fullscreen mode

Test 3: Encryption as a Service

# Enable transit engine
vault secrets enable transit

# Create an encryption key
vault write -f transit/keys/my-key

# Encrypt data
vault write transit/encrypt/my-key plaintext=@data.txt

# Decrypt it
vault write transit/decrypt/my-key ciphertext=vault:v1:...
Enter fullscreen mode Exit fullscreen mode

Monitoring and Maintenance

Check Cluster Health

# Status of all nodes
docker-compose exec vault-1 vault status

# Raft peer status
docker-compose exec vault-1 vault operator raft list-peers

# Audit logs
docker-compose logs vault-1 | grep ERROR

# System metrics
docker-compose stats
Enter fullscreen mode Exit fullscreen mode

Common Issues and Solutions

Issue Cause Solution
Node won't join Already part of a cluster Run task reset and task bootstrap
Connection refused Node not running Check with docker-compose ps
Memory locked mlock issues Check ulimits and IPC_LOCK capability

Production Considerations

1. Enable TLS/HTTPS

listener "tcp" {
  address            = "0.0.0.0:8200"
  tls_cert_file      = "/vault/config/cert.pem"
  tls_key_file       = "/vault/config/key.pem"
}
Enter fullscreen mode Exit fullscreen mode

2. Enable Audit Logging

audit {
  file {
    path = "/vault/logs/audit.log"
  }
}
Enter fullscreen mode Exit fullscreen mode

3. Configure Storage Snapshots

# Backup Raft data
vault operator raft snapshot save vault-backup.snap

# Restore from snapshot
vault operator raft snapshot restore -force vault-backup.snap
Enter fullscreen mode Exit fullscreen mode

4. Set Resource Limits

deploy:
  resources:
    limits:
      cpus: '2'
      memory: 4G
    reservations:
      cpus: '1'
      memory: 2G
Enter fullscreen mode Exit fullscreen mode

Scaling Considerations

Adding More Nodes

# Copy vault-1 directory structure
cp -r vault-1 vault-4

# Update vault-4/config/vault.hcl (change node_id)
sed -i 's/vault-1/vault-4/g' vault-4/config/vault.hcl

# Add to docker-compose.yaml and run
docker-compose up -d vault-4
Enter fullscreen mode Exit fullscreen mode

Load Balancing

upstream vault {
    server vault-1:8200;
    server vault-2:8200;
    server vault-3:8200;
}

server {
    listen 80;
    location / {
        proxy_pass http://vault;
    }
}
Enter fullscreen mode Exit fullscreen mode

Advanced: Disaster Recovery

Scenario: Complete Cluster Failure

# 1. Reset everything
task reset

# 2. Restore from backup
vault operator raft snapshot restore -force vault-backup.snap

# 3. Bring cluster back up
task bootstrap
Enter fullscreen mode Exit fullscreen mode

Scenario: Corrupted Raft State

# 1. Stop all nodes
task down

# 2. Clean data directories
rm -rf vault-*/data/raft/*

# 3. Restore from known-good backup
# Copy backed-up raft directory to all nodes

# 4. Start nodes
task up
Enter fullscreen mode Exit fullscreen mode

Conclusion

You now have a production-ready, highly-available Vault cluster with:

✅ Three-node Raft cluster for high availability

✅ Automated deployment via Docker Compose

✅ Task automation for common operations

✅ Auto-unsealing capability

✅ Health monitoring and failure detection

✅ Scalable architecture ready for growth

Resources


About the Author

This article demonstrates a practical approach to secrets management using open-source tools. For production deployments, consider consulting with security specialists to ensure compliance with your organization's security requirements.

Have questions? Share them in the comments below!


GitHub Repository: vault-docker-cluster

Top comments (2)

Collapse
 
scsoi profile image
疏影 •

Raft as integrated storage is a good choice for most Vault deployments — the integrated raft backend cuts out the external dependency (Consul / DynamoDB / etcd) and makes Vault a self-contained binary. A few things to internalize before you run it in HA:

  1. Voter count is the actual safety knob. A 5-node Raft cluster can lose 2 nodes and stay quorum; a 3-node cluster loses 1. We have seen teams run 3-node in dev and 5-node in prod, which is right. Performance leader is still 1, so adding nodes past 5 mostly buys you fault tolerance, not throughput.

  2. Performance standby vs non-voter is the partition tolerance choice. Performance standby lets a follower serve reads (token reads, KV reads), but those reads are not consistent across followers during a partition. If your read workload is heavy, run 2-3 performance standbys; if consistency matters more, run them as non-voter.

  3. Autopilot is the operational lever. vault operator raft autopilot enable with dead_server_cleanup=true + last_contact_threshold=10s will evict dead followers automatically, which avoids the scenario where a partition slowly drains quorum.

  4. Storage path on the right filesystem is the silent footgun. Raft writes append-only WAL entries; if the underlying filesystem cannot keep up with fsync, the cluster appears healthy but lag builds silently. Pinning the storage path to local NVMe is standard; using a network mount without fsync guarantees is asking for trouble.

  5. Snapshot and backup are different operations. Vault has raft snapshot save /raft snapshot restore for operational failover; for disaster recovery you want the encrypted backup operator plus the Shamir unseal keys, in a separate physical location. A backup of Raft logs without the unseal keys is useless.

  6. Cross-region Raft is the wrong move. Stretching Raft across regions turns a 50ms RTT into a 200ms one, which tanks write throughput and makes partitions more likely. For multi-region, use Vault's replication primitives (disaster recovery or performance replication), not stretched Raft.

For shops running Vault on Docker in production, the trickiest bit is usually the storage path — bind-mounting /vault/file to a network mount with sub-second fsync can silently slow writes. We have seen teams standardise on a single mount layer (ScsDriver mounts WebDAV / Aliyun OSS / S3 / NFS as a real local disk on Windows and Linux so the Raft WAL sees a stable path; failover just re-mounts at the new node). Vault does not care that the underlying blob store is object storage, as long as fsync on the WAL writes is honest.

Anyone running Vault Raft in production at >5 nodes — what motivated the size, and did autopilot save you from a real partition event?

Collapse
 
scsoi profile image
疏影 •

Vault HA with Raft is the right call for most teams — integrated storage, no external Consul dependency. A few production realities:

  1. Raft needs an odd number of voters (3 or 5). 3 is the sweet spot — 5 is only worth it for zero-downtime rolling upgrades with very long recovery times.

  2. Performance — Raft integrated storage is slower than the old Consul backend by ~15% for secret read latency. For most workloads (10s-100s of secrets/sec) this is fine. For 1000s/sec, benchmark before committing.

  3. Auto-unseal is non-optional in HA — without AWS KMS / Azure Key Vault / GCP CKMS unseal, every Vault restart is a manual ceremony. With KMS, Vault starts itself.

  4. Backup the Raft data dir — Vault operator raft snapshot save gives you an encrypted snapshot you can restore to a new cluster. Schedule daily snapshots, ship to S3 / Azure Blob with versioning. Test restore quarterly; an untested backup is a wish.

  5. Audit log to a separate path — Vault can write audit logs to a file. Mount that file as a Docker volume and ship it to your SIEM. Don't write audit logs to stdout — they get lost in container restart.

  6. Disaster recovery is snapshot-based, not replication-based — Raft clusters don't replicate to a passive standby. DR = automated snapshot + restore procedure. Document this; do not assume HA covers DR.

For ops who want to mount their Vault backup snapshots from Windows Explorer: ScsDriver