DEV Community

Cover image for vault project
Omar Ahmed
Omar Ahmed

Posted on

vault project

#devops #devsecops #vault #kubernetes

This project deploys HashiCorp Vault on Kubernetes using Helm in a production-like setup.

1. Project Structure

Create the project directory:

mkdir vault-k8s-helm-project
cd vault-k8s-helm-project

mkdir -p helm/vault
mkdir -p k8s/app
mkdir -p scripts
Enter fullscreen mode Exit fullscreen mode

Final structure:

vault/
├── helm/
│   └── vault/
│       └── values.yaml
├── k8s/
│   └── app/
│       ├── namespace.yaml
│       ├── service-account.yaml
│       └── deployment.yaml
└── scripts/
    ├── 01-install-vault.sh
    ├── 02-init-vault.sh
    ├── 03-unseal-vault.sh
    ├── 04-enable-k8s-auth.sh
    ├── 05-create-policy-and-role.sh
    ├── 06-create-secret.sh
    ├── 07-deploy-app.sh
Enter fullscreen mode Exit fullscreen mode

2. Add HashiCorp Helm Repository 

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
Enter fullscreen mode Exit fullscreen mode

Check the chart:

helm search repo hashicorp/vault
Enter fullscreen mode Exit fullscreen mode

3. Create Vault Namespace 

kubectl create namespace vault
Enter fullscreen mode Exit fullscreen mode

4. Vault Helm Values

Create the values file:

vi helm/vault/values.yaml
Enter fullscreen mode Exit fullscreen mode

Add:

global:
  enabled: true
  tlsDisable: true

injector:
  enabled: true
  replicas: 1

server:
  enabled: true

  image:
    repository: hashicorp/vault

  resources:
    requests:
      memory: 512Mi
      cpu: 250m
    limits:
      memory: 1Gi
      cpu: 1000m

  dataStorage:
    enabled: true
    size: 10Gi
    mountPath: /vault/data
    storageClass: null
    accessMode: ReadWriteOnce

  standalone:
    enabled: true
    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address     = "[::]:8200"
      }

      storage "file" {
        path = "/vault/data"
      }

  ha:
    enabled: false

  service:
    enabled: true
    type: ClusterIP

  logLevel: "info"

ui:
  enabled: true
  serviceType: ClusterIP
Enter fullscreen mode Exit fullscreen mode

5. Install Vault with Helm

Create the install script:

vi scripts/01-install-vault.sh
Enter fullscreen mode Exit fullscreen mode

Add:

#!/bin/bash
set -e

helm upgrade --install vault hashicorp/vault \
  --namespace vault \
  -f helm/vault/values.yaml

kubectl -n vault get pods
kubectl -n vault get svc
Enter fullscreen mode Exit fullscreen mode

Run:

chmod +x scripts/01-install-vault.sh
./scripts/01-install-vault.sh
Enter fullscreen mode Exit fullscreen mode

Check pods:

kubectl -n vault get pods
Enter fullscreen mode Exit fullscreen mode

Expected:

vault-0                         0/1     Running
vault-agent-injector-xxxxx      1/1     Running
Enter fullscreen mode Exit fullscreen mode

Vault pods will show 0/1 because Vault is not initialized and unsealed yet.

6. Initialize Vault

Create the init script:

vi scripts/02-init-vault.sh
Enter fullscreen mode Exit fullscreen mode

Add:

#!/bin/bash
set -e

kubectl -n vault exec vault-0 -- vault operator init \
  -key-shares=5 \
  -key-threshold=3 \
  -format=json > vault-init.json

echo "Vault initialized."
echo "IMPORTANT: Store vault-init.json securely."
cat vault-init.json
Enter fullscreen mode Exit fullscreen mode

Run:

chmod +x scripts/02-init-vault.sh
./scripts/02-init-vault.sh
Enter fullscreen mode Exit fullscreen mode

You will get:

{
  "unseal_keys_b64": [
    "...",
    "...",
    "...",
    "...",
    "..."
  ],
  "root_token": "..."
}
Enter fullscreen mode Exit fullscreen mode

Secure the file:

chmod 600 vault-init.json
Enter fullscreen mode Exit fullscreen mode

Do not commit this file.

Create .gitignore:

vi .gitignore
Enter fullscreen mode Exit fullscreen mode

Add:

vault-init.json
*.token
*.snap
Enter fullscreen mode Exit fullscreen mode

7. Unseal Vault

Create the unseal script:

vi scripts/03-unseal-vault.sh
Enter fullscreen mode Exit fullscreen mode

Add:

#!/bin/bash
set -e

KEY1=$(jq -r '.unseal_keys_b64[0]' vault-init.json)
KEY2=$(jq -r '.unseal_keys_b64[1]' vault-init.json)
KEY3=$(jq -r '.unseal_keys_b64[2]' vault-init.json)

echo "Unsealing vault-0..."

kubectl -n vault exec vault-0 -- vault operator unseal "$KEY1"
kubectl -n vault exec vault-0 -- vault operator unseal "$KEY2"
kubectl -n vault exec vault-0 -- vault operator unseal "$KEY3"

echo ""
echo "Vault status:"
kubectl -n vault exec vault-0 -- vault status
Enter fullscreen mode Exit fullscreen mode

Run:

chmod +x scripts/03-unseal-vault.sh
./scripts/03-unseal-vault.sh
Enter fullscreen mode Exit fullscreen mode

Check status:

kubectl -n vault exec vault-0 -- vault status
Enter fullscreen mode Exit fullscreen mode

Expected:

Initialized     true
Sealed          false
Enter fullscreen mode Exit fullscreen mode

8. Login to Vault

Export the root token locally:

export VAULT_ROOT_TOKEN=$(cat vault-init.json | jq -r '.root_token')
Enter fullscreen mode Exit fullscreen mode

Login inside the Vault pod:

kubectl -n vault exec -it vault-0 -- sh
Enter fullscreen mode Exit fullscreen mode

Inside the pod:

export VAULT_ADDR=http://127.0.0.1:8200
vault login
Enter fullscreen mode Exit fullscreen mode

Paste the root token.

Or run directly:

kubectl -n vault exec vault-0 -- vault login $VAULT_ROOT_TOKEN
Enter fullscreen mode Exit fullscreen mode

9. Enable Vault UI Locally

Port-forward Vault UI:

kubectl -n vault port-forward svc/vault-ui 8200:8200
Enter fullscreen mode Exit fullscreen mode

Open:

http://localhost:8200
Enter fullscreen mode Exit fullscreen mode

Login with the root token.

10. Enable KV Secrets Engine and Create Secret

Create the secret script:

vi scripts/06-create-secret.sh
Enter fullscreen mode Exit fullscreen mode

Add:

#!/bin/bash
set -e

ROOT_TOKEN=$(cat vault-init.json | jq -r '.root_token')

kubectl -n vault exec vault-0 -- sh -c "
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=$ROOT_TOKEN

vault secrets enable -path=secret kv-v2 || true
# it enables the KV (Key-Value) version 2 secrets engine and makes it accessible at the path /secret
vault secrets list

vault kv put secret/myapp/config \
  DB_HOST=postgres.default.svc.cluster.local \
  DB_PORT=5432 \
  DB_USER=myapp_user \
  DB_PASSWORD=super-secret-password \
  JWT_SECRET=my-jwt-secret

vault kv get secret/myapp/config
"
Enter fullscreen mode Exit fullscreen mode

Run:

chmod +x scripts/06-create-secret.sh
./scripts/06-create-secret.sh
Enter fullscreen mode Exit fullscreen mode

11. Enable Kubernetes Auth

Create the Kubernetes auth script:

vi scripts/04-enable-k8s-auth.sh
Enter fullscreen mode Exit fullscreen mode

Add:

#!/bin/bash
set -e

ROOT_TOKEN=$(cat vault-init.json | jq -r '.root_token')

kubectl -n vault exec vault-0 -- sh -c "
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=$ROOT_TOKEN

vault auth enable kubernetes || true

# environment variable automatically injected into every pod by Kubernetes
# kubectl -n vault exec vault-0 -- env | grep KUBERNETES
vault write auth/kubernetes/config \
  kubernetes_host=https://\$KUBERNETES_PORT_443_TCP_ADDR:443
# When pods try to authenticate using the Kubernetes auth method, here's the address of the Kubernetes API server you should use to verify their identity
"
Enter fullscreen mode Exit fullscreen mode

Run:

chmod +x scripts/04-enable-k8s-auth.sh
./scripts/04-enable-k8s-auth.sh
Enter fullscreen mode Exit fullscreen mode

12. Create Vault Policy and Kubernetes Role

Create the policy and role script:

vi scripts/05-create-policy-and-role.sh
Enter fullscreen mode Exit fullscreen mode

Add:

#!/bin/bash
set -e

ROOT_TOKEN=$(cat vault-init.json | jq -r '.root_token')

kubectl -n vault exec vault-0 -- sh -c "
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=$ROOT_TOKEN

vault policy write myapp-policy - <<EOF_POLICY
path \"secret/data/myapp/config\" {
  capabilities = [\"read\"]
}
EOF_POLICY
# myapp-policy grants permission to read the secret at secret/data/myapp/config

vault policy list
# Output: default, myapp-policy, root

vault policy read myapp-policy
# Shows the policy content back

vault write auth/kubernetes/role/myapp-role \
  bound_service_account_names=myapp-sa \
  bound_service_account_namespaces=myapp \
  policies=myapp-policy \
  ttl=24h

vault read auth/kubernetes/role/myapp-role
# Displays the configuration of the role you just created
"
Enter fullscreen mode Exit fullscreen mode

Run:

chmod +x scripts/05-create-policy-and-role.sh
./scripts/05-create-policy-and-role.sh
Enter fullscreen mode Exit fullscreen mode

Meaning:

Only pods using service account myapp-sa in namespace myapp can read secret/myapp/config.
Enter fullscreen mode Exit fullscreen mode

13. Create Example Application Namespace

Create namespace manifest:

vi k8s/app/namespace.yaml
Enter fullscreen mode Exit fullscreen mode

Add:

apiVersion: v1
kind: Namespace
metadata:
  name: myapp
Enter fullscreen mode Exit fullscreen mode

Create service account manifest:

vi k8s/app/service-account.yaml
Enter fullscreen mode Exit fullscreen mode

Add:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: myapp-sa
  namespace: myapp
Enter fullscreen mode Exit fullscreen mode

14. Example Application Using Vault Agent Injector

Create deployment manifest:

vi k8s/app/deployment.yaml
Enter fullscreen mode Exit fullscreen mode

Add:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        vault.hashicorp.com/agent-inject: "true" #inject Vault Agent container into this pod
        vault.hashicorp.com/role: "myapp-role"

        vault.hashicorp.com/agent-inject-secret-config.txt: "secret/data/myapp/config"
        # Tells the Vault Agent: "Fetch the secret at secret/data/myapp/config, and write the result to the file /vault/secrets/config.txt inside the pod."

        # The injector uses a naming convention: the suffix after agent-inject-template- becomes the filename created in /vault/secrets/
        vault.hashicorp.com/agent-inject-template-config.txt: |
          {{- with secret "secret/data/myapp/config" -}}
          DB_HOST={{ .Data.data.DB_HOST }}
          DB_PORT={{ .Data.data.DB_PORT }}
          DB_USER={{ .Data.data.DB_USER }}
          DB_PASSWORD={{ .Data.data.DB_PASSWORD }}
          JWT_SECRET={{ .Data.data.JWT_SECRET }}
          {{- end }}
    spec:
      serviceAccountName: myapp-sa
      containers:
        - name: myapp
          image: busybox:1.36
          command:
            - sh
            - -c
            - |
              echo "Starting app..."
              echo "Reading Vault secret file:"
              cat /vault/secrets/config.txt
              sleep 3600
Enter fullscreen mode Exit fullscreen mode

15. Deploy Example App

Create deploy script:

vi scripts/07-deploy-app.sh
Enter fullscreen mode Exit fullscreen mode

Add:

#!/bin/bash
set -e

kubectl apply -f k8s/app/namespace.yaml
kubectl apply -f k8s/app/service-account.yaml
kubectl apply -f k8s/app/deployment.yaml

kubectl -n myapp get pods
Enter fullscreen mode Exit fullscreen mode

Run:

chmod +x scripts/07-deploy-app.sh
./scripts/07-deploy-app.sh
Enter fullscreen mode Exit fullscreen mode

Check pod:

kubectl -n myapp get pods
Enter fullscreen mode Exit fullscreen mode

Expected:

myapp-xxxxx    2/2    Running
Enter fullscreen mode Exit fullscreen mode

Why 2/2?

Because Vault Agent sidecar was injected.

Check app logs:

kubectl -n myapp logs deploy/myapp -c myapp
Enter fullscreen mode Exit fullscreen mode

Expected:

Starting app...
Reading Vault secret file:
DB_HOST=postgres.default.svc.cluster.local
DB_PORT=5432
DB_USER=myapp_user
DB_PASSWORD=super-secret-password
JWT_SECRET=my-jwt-secret
Enter fullscreen mode Exit fullscreen mode

Check injected containers:

kubectl -n myapp describe pod <pod-name>
Enter fullscreen mode Exit fullscreen mode

You should see:

vault-agent-init
vault-agent
myapp
Enter fullscreen mode Exit fullscreen mode

16. Test

Create another service account:

kubectl -n myapp create serviceaccount wrong-sa
Enter fullscreen mode Exit fullscreen mode

Change deployment:

serviceAccountName: wrong-sa
Enter fullscreen mode Exit fullscreen mode

Apply:

kubectl apply -f k8s/app/deployment.yaml
Enter fullscreen mode Exit fullscreen mode

Vault Agent should fail to authenticate because the Vault role only allows:

service account: myapp-sa
namespace: myapp
Enter fullscreen mode Exit fullscreen mode 
kubectl get pod -n myapp
Enter fullscreen mode Exit fullscreen mode

Top comments (0)