DEV Community

Cover image for Deploying and Managing HashiCorp Vault in Kubernetes with HA and Raft Storage
Durrell Gemuh
Durrell Gemuh

Posted on

Deploying and Managing HashiCorp Vault in Kubernetes with HA and Raft Storage

#kubernetes #vault #devops #linux

Vault is a powerful secrets management tool. Running Vault on Kubernetes in HA mode with Raft backend provides resilience and scalability for secure secrets storage. This guide covers: installing Vault, setting up namespaces, deploying via Helm, joining Vault nodes, unsealing, and troubleshooting common issues.

Prerequisites

  • Kubernetes cluster accessible with kubectl
  • Helm 3 installed
  • Sufficient permissions to create namespaces and PVs
  • Basic familiarity with Vault concepts and Kubernetes

Step 1: Setting Up Namespace and Helm Repo

Create a namespace for Vault to isolate it:

kubectl create namespace vault
Enter fullscreen mode Exit fullscreen mode

Add the HashiCorp Helm repo and update:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
Enter fullscreen mode Exit fullscreen mode

Step 2: Installing Vault with Helm in HA Mode

Create a values.yaml for Vault HA using Raft storage (Integrated Storage):

injector:
  enabled: false
server:
  image:
    repository: "hashicorp/vault"
    tag: "1.9.0"
    # Overrides the default Image Pull Policy
    pullPolicy: IfNotPresent

  # Configure the Update Strategy Type for the StatefulSet
  updateStrategyType: "OnDelete"
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 256Mi
      cpu: 250m
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      config: |
        ui = true
        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }
        storage "raft" {
          path = "/vault/data"
        }
        service_registration "kubernetes" {}
  dataStorage:
    enabled: true
    size: 500Mi
Enter fullscreen mode Exit fullscreen mode

Install Vault:

helm install vault hashicorp/vault -n vault -f values.yaml
Enter fullscreen mode Exit fullscreen mode

Verify pods:

$ kubectl get pods -n vault
NAME                                           READY   STATUS    RESTARTS
vault-0                                        0/1     Running   0
vault-1                                        0/1     Running   0
vault-2                                        0/1     Running   0
Enter fullscreen mode Exit fullscreen mode

Step 3: Joining Vault Nodes and Initializing Cluster

  • Only the first Vault pod (vault-0) is initialized once.
  • Initialize Vault on vault-0: 
kubectl exec -it vault-0 -n vault -- vault operator init -key-shares=5 -key-threshold=3

Recovery Key 1: FKjt5wkzN5bUBIuR52KrPP1c2Il/f7RZdn5E+ipfNF8s
Recovery Key 2: FCzUyduESPyavh6QtqWZpdnUDKa3bEEpBHbX3NgTrCiU
Recovery Key 3: Tf7FVEpj5tdJLqQqNw/Jt0OytRI5FAZYig/yafSVz3Xg
Recovery Key 4: duLpa/6IozTOR0mkO7sp0CwmnI+1DsC6d2+oZG/A1CIZ
Recovery Key 5: pyVFs/rRFEk9rSn57Ru+KeuAQzW6eurl3j0/pS/JRpXD
Initial Root Token: s.PMOJxNxZYgvXzby5YZr9p3J1
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 keys to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
Enter fullscreen mode Exit fullscreen mode

Now, we see the Vault unsealer in action:

$ kubectl exec -it vault-0 -n vault -- vault operator unseal <unseal_key_1>
$ kubectl exec -it vault-0 -n vault -- vault operator unseal <unseal_key_2>
$ kubectl exec -it vault-0 -n vault -- vault operator unseal <unseal_key_3>

$ kubectl get pods -n vault

NAME                                           READY   STATUS    RESTARTS
vault-0                                        1/1     Running   0
vault-1                                        0/1     Running   0
vault-2                                        0/1     Running   0
Enter fullscreen mode Exit fullscreen mode

However, our replica vault-1 and vault-2 are still not ready. They are follower pods of the leader vault-0. In order to make vault-0 visible, we need to login using our Initial Root Token.

$ kubectl exec -it vault-0 -- vault login s.d0LAlSnAerb4a7d6ibkfxrZy

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key                  Value
---                  -----
token                s.d0LAlSnAerb4a7d6ibkfxrZy
token_accessor       hJEFibLTbUP4sgA8X4tMBqZ8
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
Enter fullscreen mode Exit fullscreen mode
  • Other pods (vault-1, vault-2) must join the Raft cluster. Next, let’s join vault-1 and vault-2 to Vault-0 to make the Vault setup Highly available by running commands as shown below: 
$ kubectl exec -it vault-1 -n vault -- vault operator raft join http://vault-0.vault-internal:8200
Key       Value
---       -----
Joined    true

$ kubectl exec -it vault-2 -n vault -- vault operator raft join http://vault-0.vault-internal:8200
Key       Value
---       -----
Joined    true
Enter fullscreen mode Exit fullscreen mode

Save unseal keys securely.

Step 4: Unsealing Vault Pods

Unseal all Vault pods (vault-0, vault-1, vault-2) manually with the unseal keys:

kubectl exec -it vault-1 -n vault -- vault operator unseal <unseal_key_1>
kubectl exec -it vault-1 -n vault -- vault operator unseal <unseal_key_2>
kubectl exec -it vault-1 -n vault -- vault operator unseal <unseal_key_3>

kubectl exec -it vault-2 -n vault -- vault operator unseal <unseal_key_1>
kubectl exec -it vault-2 -n vault -- vault operator unseal <unseal_key_2>
kubectl exec -it vault-2 -n vault -- vault operator unseal <unseal_key_3>
Enter fullscreen mode Exit fullscreen mode

Let’s check the pod status and this time you would be able to see them in running as well as Ready mode:

$ kubectl get pods -n vault
NAME                                           READY   STATUS    RESTARTS
vault-0                                        1/1     Running   0
vault-1                                        1/1     Running   0
vault-2                                        1/1     Running   0
Enter fullscreen mode Exit fullscreen mode

The last but important step is to verify whether the HA setup is correct or not, is by running the below command against each Vault pod and making sure that the value of HA Enabled parameters is true.

$ kubectl exec -it -n vault vault-0 -- vault status
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.9.0
Storage Type             raft
Cluster Name             vault-cluster-30882e80
Cluster ID               1afbe13a-e951-482d-266b-e31693d17e20
HA Enabled               true
HA Cluster               https://vault-0.vault-internal:8201
HA Mode                  active
Active Since             2022-01-19T04:39:37.586622342Z
Raft Committed Index     61
Raft Applied Index       61

$ kubectl exec -it -n vault vault-1 -- vault status
Key                      Value
---                      -----
…
HA Enabled               true
HA Cluster               https://vault-0.vault-internal:8201
HA Mode                  standby
Active Node Address      http://10.244.0.17:8200
Raft Committed Index     61
Raft Applied Index       61

$ kubectl exec -it -n vault vault-2 -- vault status
Key                      Value
---                      -----
HA Enabled               true
HA Cluster               https://vault-0.vault-internal:8201
HA Mode                  standby
Active Node Address      http://10.244.0.17:8200
Raft Committed Index     61
Raft Applied Index       61
Enter fullscreen mode Exit fullscreen mode

Troubleshooting Common Issues

ClusterRoleBinding Already Exists

  • Delete conflicting ClusterRoleBinding before reinstalling Helm chart: 
kubectl delete clusterrolebinding vault-server-binding
Enter fullscreen mode Exit fullscreen mode

Helm Upgrade Errors ("no deployed releases")

  • Uninstall previous release from default namespace: 
helm uninstall vault -n default
Enter fullscreen mode Exit fullscreen mode
  • Then install again in the proper namespace.

Pods Not Unsealing or Ready

  • Check pod logs for errors: 
kubectl logs vault-1 -n vault
Enter fullscreen mode Exit fullscreen mode
  • Verify PVCs are bound and healthy: 
kubectl get pvc -n vault
Enter fullscreen mode Exit fullscreen mode
  • If corrupted PVC, delete PVC and pod, then rejoin: 
kubectl delete pvc data-vault-1 -n vault
kubectl delete pod vault-1 -n vault
kubectl exec -it vault-1 -n vault -- vault operator raft join http://vault-0.vault-internal:8200
Enter fullscreen mode Exit fullscreen mode
  • Unseal vault-1 after join with the same unseal keys.

PVC Stuck in Terminating

  • Remove finalizers to force delete: 
kubectl patch pvc data-vault-1 -n vault -p '{"metadata":{"finalizers":[]}}' --type=merge
Enter fullscreen mode Exit fullscreen mode

Removing Corrupted Raft Node

  • Remove node from cluster: 
kubectl exec -it vault-0 -n vault -- vault operator raft remove-peer <node-id>
Enter fullscreen mode Exit fullscreen mode
  • Delete PVC and pod, then re-join as new.

Conclusion

Deploying Vault in Kubernetes with HA and Raft backend provides secure secret storage with scalability. Proper initialization, joining, and unsealing procedures are critical. Troubleshooting involves managing PVC health, pod status, and cluster raft membership carefully.

Up Next: Vault Web UI - Accessing and viewing vault on UI

Top comments (0)