DEV Community

Cover image for vault project
Omar Ahmed
Omar Ahmed

Posted on Edited on

vault project

#devops #devsecops #vault #kubernetes

Read this first

Vault on Kubernetes

This project deploys HashiCorp Vault on Kubernetes using Helm in a production-like setup.

1. Project Structure

Create the project directory:

mkdir vault-k8s-helm-project
cd vault-k8s-helm-project

mkdir -p helm/vault
mkdir -p k8s/app
Enter fullscreen mode Exit fullscreen mode

Final structure:

vault/
├── helm/
│   └── vault/
│       └── values.yaml
├── k8s/
│   └── app/
│       ├── namespace.yaml
│       ├── service-account.yaml
│       └── deployment.yaml
Enter fullscreen mode Exit fullscreen mode

2. Add HashiCorp Helm Repository 

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
Enter fullscreen mode Exit fullscreen mode

Check the chart:

helm search repo hashicorp/vault
Enter fullscreen mode Exit fullscreen mode

3. Create Vault Namespace 

kubectl create namespace vault
Enter fullscreen mode Exit fullscreen mode

4. Vault Helm Values

Create the values file:

cat > helm/vault/values.yaml <<'YAML'
server:
  dev:
    enabled: false
  standalone:
    enabled: true
    config: |
      ui = true
      listener "tcp" {
        address         = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_disable     = 1
      }
      storage "raft" {
        path = "/vault/data"
      }
  dataStorage:
    enabled: true
    size: 1Gi

ui:
  enabled: true
  serviceType: ClusterIP
YAML
Enter fullscreen mode Exit fullscreen mode

5. Install Vault with Helm 


helm upgrade --install vault hashicorp/vault \
  --namespace vault \
  -f helm/vault/values.yaml

kubectl -n vault get pods
kubectl -n vault get svc
Enter fullscreen mode Exit fullscreen mode

Expected:

vault-0                         0/1     Running
vault-agent-injector-xxxxx      1/1     Running
Enter fullscreen mode Exit fullscreen mode

Vault pods will show 0/1 because Vault is not initialized and unsealed yet.

6. Initialize Vault 

kubectl -n vault exec vault-0 -- vault operator init \
  -key-shares=5 \
  -key-threshold=3 \
  -format=json > vault-init.json

# Initializes a fresh Vault server for the first time, 

cat vault-init.json
Enter fullscreen mode Exit fullscreen mode

You will get:

{
  "unseal_keys_b64": [
    "...",
    "...",
    "...",
    "...",
    "..."
  ],
  "root_token": "..."
}
Enter fullscreen mode Exit fullscreen mode

7. Unseal Vault 

#!/bin/bash

KEY1=$(jq -r '.unseal_keys_b64[0]' vault-init.json)
KEY2=$(jq -r '.unseal_keys_b64[1]' vault-init.json)
KEY3=$(jq -r '.unseal_keys_b64[2]' vault-init.json)

echo "Unsealing vault-0..."

kubectl -n vault exec vault-0 -- vault operator unseal "$KEY1"
kubectl -n vault exec vault-0 -- vault operator unseal "$KEY2"
kubectl -n vault exec vault-0 -- vault operator unseal "$KEY3"

kubectl -n vault exec vault-0 -- vault status
Enter fullscreen mode Exit fullscreen mode

Expected:

Initialized     true
Sealed          false
Enter fullscreen mode Exit fullscreen mode

8. Login to Vault

Export the root token locally:

VAULT_ROOT_TOKEN=$(cat vault-init.json | jq -r '.root_token')
echo $VAULT_ROOT_TOKEN
kubectl -n vault exec vault-0 -- vault login "$VAULT_ROOT_TOKEN"
# Authenticates you to Vault using a token
Enter fullscreen mode Exit fullscreen mode

9. Enable Vault UI Locally

Port-forward Vault UI:

kubectl -n vault port-forward svc/vault-ui 8200:8200
Enter fullscreen mode Exit fullscreen mode

Open:

http://localhost:8200
Enter fullscreen mode Exit fullscreen mode

Login with the root token.

10. Enable KV Secrets Engine and Create Secret 


ROOT_TOKEN=$(cat vault-init.json | jq -r '.root_token')

kubectl -n vault exec vault-0 -- sh -c "
vault secrets enable -path=secret kv-v2 || true
# it enables the KV (Key-Value) version 2 secrets engine and makes it accessible at the path /secret

echo '##Secrets List'
vault secrets list

vault kv put secret/myapp/config \
  DB_HOST=postgres.default.svc.cluster.local \
  DB_PORT=5432 \
  DB_USER=myapp_user \
  DB_PASSWORD=super-secret-password \
  JWT_SECRET=my-jwt-secret

echo '##Get Secrets'
vault kv get secret/myapp/config
"
Enter fullscreen mode Exit fullscreen mode

11. Authenticate with Kubernetes 

kubectl create ns myapps

# JWT Token
kubectl create sa vault-reviewer -n myapps
TOKEN_REVIEW_JWT=$(kubectl -n myapps create token vault-reviewer --duration=24h)

cat > vault-reviewer-binding.yaml <<'YAML'
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-reviewer-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-reviewer
  namespace: myapps
YAML

kubectl apply -f vault-reviewer-binding.yaml

# Get Kubernetes API Info
KUBE_HOST=$(kubectl config view --raw -o=jsonpath='{.clusters[0].cluster.server}')

# Kubernetes CA Cert
KUBE_CA_CERT=$(kubectl config view --raw \
  -o=jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d)

kubectl exec -n vault -it vault-0 -- vault auth enable kubernetes

kubectl exec -n vault -it vault-0 -- vault write auth/kubernetes/config \
  token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
  kubernetes_host="$KUBE_HOST" \
  kubernetes_ca_cert="$KUBE_CA_CERT"
Enter fullscreen mode Exit fullscreen mode

12. Create Vault Policy and Kubernetes Role

Create the policy and role script:

kubectl create ns myapp
kubectl create sa myapp -n myapp

kubectl -n vault exec vault-0 -- sh -c "
vault policy write myapp-policy - <<EOF_POLICY
path \"secret/data/myapp/config\" {
  capabilities = [\"read\"]
}
EOF_POLICY
# myapp-policy grants permission to read the secret at secret/data/myapp/config

echo '## List Policies'
vault policy list

echo '## Read Specific Policy'
vault policy read myapp-policy
# Shows the policy content back

vault write auth/kubernetes/role/myapp-role \
  bound_service_account_names=myapp \
  bound_service_account_namespaces=myapp \
  policies=myapp-policy \
  ttl=24h

echo '## Read Role'
vault read auth/kubernetes/role/myapp-role
# Displays the configuration of the role you just created
"
Enter fullscreen mode Exit fullscreen mode

Meaning:

Only pods using service account myapp-sa in namespace myapp can read secret/myapp/config.
Enter fullscreen mode Exit fullscreen mode

13. Example Application Using Vault Agent Injector

Create deployment manifest:

touch k8s/app/deployment.yaml
Enter fullscreen mode Exit fullscreen mode

Add:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        vault.hashicorp.com/agent-inject: "true" #inject Vault Agent container into this pod
        vault.hashicorp.com/role: "myapp-role"

        vault.hashicorp.com/agent-inject-secret-config.txt: "secret/data/myapp/config"
        # Tells the Vault Agent: "Fetch the secret at secret/data/myapp/config, and write the result to the file /vault/secrets/config.txt inside the pod."

        # The injector uses a naming convention: the suffix after agent-inject-template- becomes the filename created in /vault/secrets/
        vault.hashicorp.com/agent-inject-template-config.txt: |
          {{- with secret "secret/data/myapp/config" -}}
          DB_HOST={{ .Data.data.DB_HOST }}
          DB_PORT={{ .Data.data.DB_PORT }}
          DB_USER={{ .Data.data.DB_USER }}
          DB_PASSWORD={{ .Data.data.DB_PASSWORD }}
          JWT_SECRET={{ .Data.data.JWT_SECRET }}
          {{- end }}
    spec:
      serviceAccountName: myapp-sa
      containers:
        - name: myapp
          image: busybox:1.36
          command:
            - sh
            - -c
            - |
              echo "Starting app..."
              echo "Reading Vault secret file:"
              cat /vault/secrets/config.txt
              sleep 3600
Enter fullscreen mode Exit fullscreen mode

14. Deploy Example App 

kubectl apply -f k8s/app/deployment.yaml
kubectl -n myapp get pods
Enter fullscreen mode Exit fullscreen mode

Expected:

NAME                     READY   STATUS     RESTARTS   AGE
myapp-59b648f545-t97kb   0/2     Init:0/1   0          2m44s

## Then 
myapp-59b648f545-tf48g   2/2     Running           0          13s
Enter fullscreen mode Exit fullscreen mode

Why 2/2?

Because Vault Agent sidecar was injected.

Check app logs:

kubectl -n myapp logs deploy/myapp -c myapp
Enter fullscreen mode Exit fullscreen mode

Expected:

Starting app...
Reading Vault secret file:
DB_HOST=postgres.default.svc.cluster.local
DB_PORT=5432
DB_USER=myapp_user
DB_PASSWORD=super-secret-password
JWT_SECRET=my-jwt-secret
Enter fullscreen mode Exit fullscreen mode

Check injected containers:

kubectl -n myapp describe pod <pod-name>
Enter fullscreen mode Exit fullscreen mode

You should see:

vault-agent-init   (init container)
vault-agent        (sidecar container)
myapp
Enter fullscreen mode Exit fullscreen mode

🧩 vault-agent-init (Init Container)

Fetch the secret ONCE before your app starts

  1. Pod starts
  2. vault-agent-init runs FIRST
  3. Authenticates to Vault (using ServiceAccount JWT)
  4. Reads secret from Vault
  5. Writes file → /vault/secrets/config.txt
  6. Exits

🧩 vault-agent (Sidecar Container)

Keep the secret UPDATED while the app is running

  1. Runs alongside your app
  2. Authenticates to Vault
  3. Watches the secret
  4. If secret changes → rewrites file
  5. Renews Vault token automatically

15. Test

Create another service account:

kubectl -n myapp create serviceaccount wrong-sa
Enter fullscreen mode Exit fullscreen mode

Change deployment:

serviceAccountName: wrong-sa
Enter fullscreen mode Exit fullscreen mode

Apply:

kubectl apply -f k8s/app/deployment.yaml
Enter fullscreen mode Exit fullscreen mode

Vault Agent should fail to authenticate because the Vault role only allows:

service account: myapp-sa
namespace: myapp
Enter fullscreen mode Exit fullscreen mode 
kubectl get pod -n myapp
Enter fullscreen mode Exit fullscreen mode

Top comments (0)