DEV Community

Cover image for Building a Production-Grade Vault PKI for a Local kubeadm Cluster Without the Shortcuts.
Noah Makau
Noah Makau

Posted on

Building a Production-Grade Vault PKI for a Local kubeadm Cluster Without the Shortcuts.

#platformengineering #kubernetes #tutorial #cks

Part 3 of 7 — The Mac Kubernetes Lab: A Production-Mirror Setup from Scratch.

Previously in Part 2: Cluster 1- the OrbStack-native daily driver is up with Istio, Vault dev mode, and Crossplane. Now we start the harder half: a four-VM kubeadm cluster that mirrors a real production EKS setup.

Most kubeadm tutorials skip the part that matters most for production parity. They let kubeadm generate self-signed certificates and call it done. The cluster works, but the certificate hierarchy looks nothing like what you actually run in production, where leaf certs come from an intermediate CA that’s signed by a root CA, all managed by HashiCorp Vault.

This article walks through the boring-but-important version. Four OrbStack VMs, real networking, and a 3-tier Vault PKI that will sign every certificate in the Kubernetes cluster we boot up in Part 4. By the end of it, the mental model in your lab matches the mental model in production — short-lived admin certs, role-based issuance, full audit trail.

Why a separate VM cluster at all?

The OrbStack-native cluster from Part 2 is excellent for daily iteration, but it has a hard limitation: it’s single-node. You can’t test multi-node behaviours; node affinity, pod disruption budgets, rolling upgrades across nodes, or the kind of disk-pressure incidents that happen when one node misbehaves.

Cluster 2 is built for that. Four VMs, a genuine kubeadm-bootstrapped cluster, and a certificate authority managed by Vault, the same PKI approach used in many enterprise Kubernetes deployments. It’s also the cluster I use for CKS exam preparation, because the exam gives you something very similar.

VM cluster topology — vault VM acts as PKI CA, cp01 is the control plane, worker01 and worker02 are worker nodes.

Creating the VMs.

OrbStack VMs are Ubuntu Noble (24.04) on ARM64. They boot in under three seconds and share memory with the host - they only consume what they actually need.

# 💻 Mac
orb create ubuntu:noble vault
orb create ubuntu:noble cp01
orb create ubuntu:noble worker01
orb create ubuntu:noble worker02

# Verify
orb list
Enter fullscreen mode Exit fullscreen mode

Getting into any VM is short:

# 💻 Mac
ssh vault@orb
ssh cp01@orb
ssh worker01@orb
ssh worker02@orb
Enter fullscreen mode Exit fullscreen mode

No key management. No IP lookup. OrbStack handles SSH automatically — which alone is worth the switch for me, since I’ve spent too much of my life copy-pasting SSH commands around.

Record the IPs: 

# 💻 Mac
for vm in vault cp01 worker01 worker02; do
  echo "$vm: $(orb run -m $vm hostname -I | awk '{print $1}')"
done
Enter fullscreen mode Exit fullscreen mode

Output looks like:

vault: 192.168.139.100
cp01: 192.168.139.101
worker01: 192.168.139.102
worker02: 192.168.139.103
Enter fullscreen mode Exit fullscreen mode

Set these on your Mac and persist them on each VM:

# 💻 Mac
export VAULT_IP=192.168.139.100
export CP_IP=192.168.139.101
export W1_IP=192.168.139.102
export W2_IP=192.168.139.103
Enter fullscreen mode Exit fullscreen mode 
# 🖥️ VM: vault / cp01 / worker01 / worker02 (run on each)
cat >> ~/.bashrc <<EOF
export VAULT_IP=<vault-ip>
export CP_IP=<cp01-ip>
export W1_IP=<worker01-ip>
export W2_IP=<worker02-ip>
EOF
source ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

Why persist to .bashrc? OrbStack VM shell sessions don't share environment variables. Every time you SSH into a VM, you start with a clean environment. Persisting the IPs to .bashrc means they're always available without re-exporting. This is the single biggest day-to-day friction in the entire lab — I'll come back to it in Part 7.

/etc/hosts on all VMs:

Each VM needs to resolve the others by hostname:

# 🖥️ VM: vault / cp01 / worker01 / worker02 (run on each)
sudo tee -a /etc/hosts <<EOF
$VAULT_IP    vault vault.lab.local
$CP_IP       cp01 cp01.lab.local kubernetes
$W1_IP       worker01 worker01.lab.local
$W2_IP       worker02 worker02.lab.local
EOF
Enter fullscreen mode Exit fullscreen mode

Why Vault PKI, not kubeadm’s defaults?

This is the part most local-cluster tutorials skip. They use whatever self-signed certificates kubeadm generates automatically. That works, but it has nothing in common with how certificates are managed in a real enterprise cluster.

In production, I use Vault PKI because it gives me:

  • A proper certificate hierarchy — Root CA → Intermediate CA → leaf certificates.
  • Short-lived credentials — admin certificates with one- or two-hour TTLs instead of long-lived kubeconfig credentials.
  • Auditability — every certificate issuance is logged in Vault.
  • Role-based issuance — separate roles for kube-apiserver, kubelet, etcd, and admin, each with its own constraints. Setting it up locally means the mental model from the lab matches production. It also means when something goes wrong with cert rotation in production, you’ve already debugged it once on your laptop, where the stakes are zero.

3-tier PKI hierarchy — Root CA signs the Intermediate CA, which issues all cluster certificates.<br>

Installing Vault on the vault VM.

The minimal OrbStack Ubuntu image doesn’t include gpg by default. This trips people up. Adding the HashiCorp repo before installing gpg leaves a broken, unsigned repo entry that blocks every future apt update call. Always install gpg first and clean up any stale entries from a previous attempt.

# 🖥️ VM: vault

# Remove any stale repo entry from a previous attempt
sudo rm -f /etc/apt/sources.list.d/hashicorp.list

# Install prerequisites first
sudo apt update && sudo apt install -y gpg curl apt-transport-https ca-certificates jq

# Import HashiCorp GPG key
curl -fsSL https://apt.releases.hashicorp.com/gpg | \
  sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

# Add repo with signed-by reference
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
  https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
  | sudo tee /etc/apt/sources.list.d/hashicorp.list

sudo apt update && sudo apt install -y vault
Enter fullscreen mode Exit fullscreen mode

Configure and start Vault: 

# 🖥️ VM: vault

sudo mkdir -p /opt/vault/data

sudo chown -R vault:vault /opt/vault/data

sudo tee /etc/vault.d/vault.hcl <<EOF
storage "file" {
  path = "/opt/vault/data"
}
listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = true
}
ui       = true
api_addr = "http://vault.lab.local:8200"
EOF

sudo chown vault:vault /etc/vault.d/vault.hcl

sudo systemctl enable vault && sudo systemctl start vault

export VAULT_ADDR='http://127.0.0.1:8200'

vault status
Enter fullscreen mode Exit fullscreen mode

Initialize and unseal: 

# 🖥️ VM: vault

# Save to home dir - writing to /root gives "permission denied"
vault operator init -key-shares=1 -key-threshold=1 > ~/vault-init.txt

export VAULT_UNSEAL_KEY=$(grep 'Unseal Key 1' ~/vault-init.txt | awk '{print $NF}')
export VAULT_ROOT_TOKEN=$(grep 'Initial Root Token' ~/vault-init.txt | awk '{print $NF}')

vault operator unseal $VAULT_UNSEAL_KEY
vault login $VAULT_ROOT_TOKEN

# Persist so these survive session restarts
echo "export VAULT_ADDR='http://127.0.0.1:8200'" >> ~/.bashrc
echo "export VAULT_TOKEN=$(grep 'Initial Root Token' ~/vault-init.txt | awk '{print $NF}')" >> ~/.bashrc

source ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

Save ~/vault-init.txt carefully. With a single key share, this file contains everything needed to unseal and access Vault. In production, use multiple key shares and distribute them. For a lab, one share is fine.

Building the 3-tier PKI:

 Root CA:

The Root CA signs nothing directly except the Intermediate CA. In a real production setup, the root key would be kept offline. Here we keep it in the Vault, but never use it for anything else after the initial intermediate signing.

# 🖥️ VM: vault

vault secrets enable -path=pki pki
vault secrets tune -max-lease-ttl=87600h pki   # 10 years

vault write -field=certificate pki/root/generate/internal \
  common_name="Lab Root CA" ttl=87600h > /tmp/root-ca.crt

vault write pki/config/urls \
  issuing_certificates="http://vault.lab.local:8200/v1/pki/ca" \
  crl_distribution_points="http://vault.lab.local:8200/v1/pki/crl"
Enter fullscreen mode Exit fullscreen mode

K8s Intermediate CA: the part most guides get wrong:

Here is where most guides go wrong. The default intermediate CA type in Vault is internal; the private key is generated inside Vault and never leaves. That's great for security, but kubeadm needs the CA private key on disk at /etc/kubernetes/pki/ca.key to sign cluster certificates.
You must use the exported type. The private key is returned only once, at generation time, so save the full JSON response before extracting anything from it. If you lose this output, you can't recover the key from Vault later. I learned this the hard way.

# 🖥️ VM: vault

vault secrets enable -path=pki_k8s pki
vault secrets tune -max-lease-ttl=43800h pki_k8s   # 5 years

# CRITICAL: use 'exported' not 'internal'
# Save the FULL JSON - private key is only returned at generation time
vault write -format=json pki_k8s/intermediate/generate/exported \
  common_name="Lab K8s Intermediate CA" \
  | tee /tmp/intermediate-full.json \
  | jq -r '.data.csr' > /tmp/k8s-intermediate.csr

# Extract and save the private key immediately
jq -r '.data.private_key' /tmp/intermediate-full.json > /tmp/ca.key

# Verify - must show -----BEGIN RSA PRIVATE KEY-----
cat /tmp/ca.key
Enter fullscreen mode Exit fullscreen mode

Sign the intermediate with the Root CA and import it back:

# 🖥️ VM: vault

vault write -format=json pki/root/sign-intermediate \
  csr=@/tmp/k8s-intermediate.csr format=pem_bundle ttl=43800h \
  | jq -r '.data.certificate' > /tmp/k8s-intermediate-signed.pem

vault write pki_k8s/intermediate/set-signed \
  certificate=@/tmp/k8s-intermediate-signed.pem

# Configure AIA URLs - resolves the "authority information access" warning
vault write pki_k8s/config/urls \
  issuing_certificates="http://vault.lab.local:8200/v1/pki_k8s/ca" \
  crl_distribution_points="http://vault.lab.local:8200/v1/pki_k8s/crl"
Enter fullscreen mode Exit fullscreen mode

Roles:

Roles define what certificates each component of the cluster can request:

# 🖥️ VM: vault

# API server - SANs for all DNS names and IPs the apiserver answers on
vault write pki_k8s/roles/kube-apiserver \
  allowed_domains="kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster.local,cp01,cp01.lab.local" \
  allow_bare_domains=true allow_subdomains=true allow_ip_sans=true max_ttl=8760h

# Kubelet - nodes in the system:nodes group
vault write pki_k8s/roles/kubelet \
  allowed_domains="system:node" allow_bare_domains=true \
  organization="system:nodes" max_ttl=8760h

# Admin - short-lived, 2 hour maximum (CKS best practice).
# This forces regular credential rotation and prevents long-lived kubeconfig files.
vault write pki_k8s/roles/admin \
  allowed_domains="kubernetes-admin" allow_bare_domains=true \
  organization="system:masters" server_flag=false client_flag=true max_ttl=2h

# etcd
vault write pki_k8s/roles/etcd \
  allow_any_name=true max_ttl=8760h

# Policy for cluster bootstrap
vault policy write k8s-pki - <<EOF
path "pki_k8s/issue/*" { capabilities = ["create","update"] }
path "pki_k8s/sign/*"  { capabilities = ["create","update"] }
path "pki_k8s/ca"      { capabilities = ["read"] }
path "pki_k8s/ca/pem"  { capabilities = ["read"] }
path "pki_k8s/crl"     { capabilities = ["read"] }
EOF
Enter fullscreen mode Exit fullscreen mode

Distributing the CA cert and key to cp01

kubeadm looks for /etc/kubernetes/pki/ca.crt and /etc/kubernetes/pki/ca.key before init. If both files are present, it uses them as the cluster CA instead of generating new self-signed certificates, which is exactly what we want.

# 🖥️ VM: vault — issue an apiserver cert to extract the CA cert
export CP_IP=<cp01-ip>   # set explicitly — does not persist across sessions
echo "CP_IP=$CP_IP"

vault write -format=json pki_k8s/issue/kube-apiserver \
  common_name="kubernetes" \
  alt_names="kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster.local,cp01" \
  ip_sans="$CP_IP,127.0.0.1,10.96.0.1" ttl=8760h > /tmp/apiserver-cert.json
jq -r '.data.issuing_ca' /tmp/apiserver-cert.json > /tmp/ca.crt
Enter fullscreen mode Exit fullscreen mode 
# 💻 Mac — pull both files and push them to cp01
orb run -m vault cat /tmp/ca.crt > /tmp/lab-ca.crt
orb run -m vault cat /tmp/ca.key > /tmp/lab-ca.key

orb run -m cp01 sudo mkdir -p /etc/kubernetes/pki
cat /tmp/lab-ca.crt | orb run -m cp01 sudo tee /etc/kubernetes/pki/ca.crt
cat /tmp/lab-ca.key | orb run -m cp01 sudo tee /etc/kubernetes/pki/ca.key

# kubeadm requires strict permissions on ca.key
orb run -m cp01 sudo chmod 600 /etc/kubernetes/pki/ca.key
orb run -m cp01 sudo ls -la /etc/kubernetes/pki/
# -rw-r--r-- ca.crt
# -rw------- ca.key
Enter fullscreen mode Exit fullscreen mode

Why orb run -m vault cat instead of scp or orb shell? When you need file content on the Mac, orb run pipes stdout directly. orb shell opens an interactive session and SCP requires setting up keys. For simple file reads, orb run -m cat > local-file is the cleanest approach.

A note on /tmp

/tmp on OrbStack VMs does not persist across reboots. The CA cert and key you just extracted will be gone the next time you restart the vault VM. That's fine — you can always regenerate them from Vault:

# 🖥️ VM: vault — regenerate CA cert after reboot
vault read -field=certificate pki_k8s/issuer/default > /tmp/lab-ca.crt
Enter fullscreen mode Exit fullscreen mode

The private key (/tmp/ca.key) was saved from the exported generation step. If you need it again, re-export from ~/intermediate-full.json, which is in your home directory and does persist:

# 🖥️ VM: vault
jq -r '.data.private_key' ~/intermediate-full.json > /tmp/ca.key
Enter fullscreen mode Exit fullscreen mode

Where we are

At this point:

  • ✅ Four OrbStack VMs created and networked
  • ✅ /etc/hosts configured on all VMs for hostname resolution
  • ✅ Vault installed, initialised, and unsealed on the vault VM
  • ✅ 3-tier PKI: Root CA → K8s Intermediate CA (exported type)
  • ✅ Roles for kube-apiserver, kubelet, etcd, and short-lived admin
  • ✅ ca.crt and ca.key placed on cp01 ready for kubeadm In Part 4, we run kubeadm init and hit the most interesting problem in this entire setup — why Calico works on M4 but quietly fails on M1, and how Cilium's eBPF dataplane solves it.

← Part 2: One Command, One Working Kubernetes Cluster! Building My Daily-Driver Lab on OrbStack | Part 4: Same Cluster, Different Mac: A Debugging Story About Unprivileged LXC Containers, iptables, and Why Cilium Replaces kube-proxy →

I’m Noah Makau, a DevSecOps engineer based in Nairobi. I run a small DevOps consultancy and hold CKA, CKAD, and the AWS Solutions Architect Professional certifications, currently preparing for CKS. I write about Kubernetes, Vault, Crossplane, and the day-to-day of running platforms that actually have to stay up.

originally published at blog.arkilasystems.com

Top comments (0)