DEV Community

Dinh Doan Van Bien
Dinh Doan Van Bien

Posted on

Part 5 — Vault, and the afternoon I deleted everything

#supabase #selfhosted #docker #devops

Part 5 of 7 — Self-hosting Supabase: a learning journey

Also available in French: Partie 5 — Vault, et l'après-midi où j'ai tout effacé

I want to tell you about the afternoon I replaced all my Supabase secrets with the word change_me.

My Postgres password: change_me. My JWT secret: change_me. My service role key: change_me. Everything.

The services started. The health checks passed (Traefik only checks HTTP status codes). Then every API call started failing with authentication errors. I SSH'd in, checked the running environment, and saw it immediately. One wrong command had replaced the entire secret store with a single key-value pair.

Not a rogue script. The Supabase Docker Compose template uses ${POSTGRES_PASSWORD:-change_me} as a fallback default for every secret variable — a placeholder that is supposed to be overridden before deployment. When my fetch script regenerated the .env from Vault, it found only one key. Everything else fell back to the template default. I explain exactly how this happened in the next section.

The good news is that HashiCorp Vault keeps a full version history. I recovered everything from version 7 of my secret. But it was a stressful 20 minutes, and I will explain exactly how to not do this mistake.

Why bother with Vault

The alternative to Vault is keeping secrets in .env files on the server. This works, but it has problems.

The obvious one: if you ever commit a .env file to git by mistake, your credentials are in version history permanently. Deleting the file does not help. The history is there.

The less obvious one: with two projects running, you have two .env files. You need a system for rotating secrets and for knowing which version of which secret was deployed at what time. Flat files do not give you any of that.

HashiCorp Vault solves all of it. Secrets are encrypted at rest. Access is controlled by tokens with specific permissions. Every change is versioned. You can recover any previous version of any secret.

We run Vault on the same server as our Docker stacks, bound only to localhost. It is never accessible from the internet.

Installing Vault

Install jq first. The fetch script uses it to parse Vault's JSON output:

apt install jq -y
Enter fullscreen mode Exit fullscreen mode 
wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor \
  -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
  https://apt.releases.hashicorp.com $(lsb_release -cs) main" \
  | tee /etc/apt/sources.list.d/hashicorp.list

apt update && apt install vault -y
Enter fullscreen mode Exit fullscreen mode

Create the configuration:

mkdir -p /etc/vault
cat > /etc/vault/config.hcl << 'EOF'
ui             = false
disable_mlock  = true

storage "file" {
  path = "/opt/vault/data"
}

listener "tcp" {
  address       = "127.0.0.1:8200"
  tls_cert_file = "/etc/vault/tls/vault.crt"
  tls_key_file  = "/etc/vault/tls/vault.key"
}
EOF
Enter fullscreen mode Exit fullscreen mode

Two things to explain here.

disable_mlock = true: By default, Vault locks all its memory pages to prevent secrets from being swapped to disk. This is a good idea in principle, but it means Vault reserves about 376 MB of RAM even at idle, since all pages are pinned. On our 4 GB server with 17 running containers, that is a sensible thing to disable. With disable_mlock = true, Vault uses about 140 MB. For a single node without hardware security modules, this is an acceptable trade-off. I do not think it is worth using a cloud HSM for a hobby setup.

address = "127.0.0.1:8200": Vault listens only on localhost. It is accessible only from the server itself, not from the network.

Vault requires TLS even for localhost connections. Generate a self-signed certificate:

mkdir -p /etc/vault/tls
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 \
  -keyout /etc/vault/tls/vault.key \
  -out /etc/vault/tls/vault.crt \
  -subj "/CN=vault-localhost" \
  -addext "subjectAltName=IP:127.0.0.1"
Enter fullscreen mode Exit fullscreen mode

Create the systemd unit and start Vault:

cat > /etc/systemd/system/vault.service << 'EOF'
[Unit]
Description=HashiCorp Vault
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/bin/vault server -config=/etc/vault/config.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable vault
systemctl start vault
Enter fullscreen mode Exit fullscreen mode

Initialize Vault:

vault operator init -key-shares=1 -key-threshold=1
Enter fullscreen mode Exit fullscreen mode

This prints an unseal key and a root token. Save them both somewhere safe and offline. A password manager's secure note is fine. If you lose the unseal key, your encrypted data is permanently inaccessible. There is no recovery path.

Unseal Vault:

VAULT_ADDR=https://127.0.0.1:8200 VAULT_SKIP_VERIFY=true \
  vault operator unseal YOUR_UNSEAL_KEY
Enter fullscreen mode Exit fullscreen mode

Storing secrets

Enable the KV version 2 secrets engine:

export VAULT_ADDR=https://127.0.0.1:8200
export VAULT_SKIP_VERIFY=true
export VAULT_TOKEN=YOUR_ROOT_TOKEN

vault secrets enable -path=secret kv-v2
Enter fullscreen mode Exit fullscreen mode

Store project1's secrets:

vault kv put secret/project1 \
  POSTGRES_PASSWORD="$(openssl rand -hex 16)" \
  JWT_SECRET="$(openssl rand -hex 32)" \
  SUPABASE_ANON_KEY="your-anon-jwt" \
  SUPABASE_SERVICE_ROLE_KEY="your-service-role-jwt" \
  API_EXTERNAL_URL="https://kong.project1.yourdomain.com" \
  GOTRUE_EXTERNAL_URL="https://kong.project1.yourdomain.com" \
  SITE_URL="https://kong.project1.yourdomain.com" \
  DB_ENC_KEY="supabaserealtime" \
  GOTRUE_MAILER_AUTOCONFIRM="false" \
  SECRET_KEY_BASE="$(openssl rand -hex 64)" \
  PG_META_CRYPTO_KEY="$(openssl rand -hex 16)"
Enter fullscreen mode Exit fullscreen mode

The mistake: put versus patch

KV v2 has two commands for writing secrets:

vault kv put replaces the entire secret with exactly what you specify. If you run vault kv put secret/project1 GOTRUE_MAILER_AUTOCONFIRM=true, the result is a secret containing exactly one key: GOTRUE_MAILER_AUTOCONFIRM. Everything else is gone from the current version.

vault kv patch merges new keys into the existing secret. Running vault kv patch secret/project1 GOTRUE_MAILER_AUTOCONFIRM=true adds the key while keeping everything else intact.

I needed to add GOTRUE_MAILER_AUTOCONFIRM=true to enable email autoconfirm for a load test. I used vault kv put. Every other key was wiped from the current version.


⚠️ Use vault kv patch when adding or updating individual keys. Reserve vault kv put for when you intentionally want to replace the entire secret.

Recovery from version history

KV v2 keeps every previous version of a secret. List the versions:

vault kv metadata get secret/project1
Enter fullscreen mode Exit fullscreen mode

Read a specific version:

vault kv get -version=7 secret/project1
Enter fullscreen mode Exit fullscreen mode

I found all my original secrets in version 7 and used vault kv patch to restore them into the current version. Twenty minutes of stress, entirely recoverable.

This version history is not just a nice-to-have. It is the reason to use KV v2 rather than KV v1.

Fetching secrets for deployment

We use a script that reads from Vault and writes a .env file:

#!/usr/bin/env bash
PROJECT=$1
VAULT_ADDR=https://127.0.0.1:8200
VAULT_SKIP_VERIFY=true
VAULT_TOKEN=$(cat /root/${PROJECT}-token.txt)

vault kv get -format=json secret/${PROJECT} \
  | jq -r '.data.data | to_entries[] | "\(.key)=\(.value)"' \
  > /root/supabase-vps-cluster/instances/${PROJECT}/.env
Enter fullscreen mode Exit fullscreen mode

The deployment sequence becomes:

bash scripts/fetch-env-from-vault.sh project1
set -a && source instances/project1/.env && set +a
docker stack deploy -c instances/project1/docker-compose.yml project1
Enter fullscreen mode Exit fullscreen mode

Per-project tokens

The root Vault token has unrestricted access to everything. We do not use it for deployments. Instead, we create a policy that grants read-only access to a single project's secrets:

# vault-policy-project1.hcl
path "secret/data/project1" {
  capabilities = ["read", "list"]
}
path "secret/metadata/project1" {
  capabilities = ["read", "list"]
}
Enter fullscreen mode Exit fullscreen mode 
vault policy write project1-readonly vault-policy-project1.hcl
vault token create -policy=project1-readonly -ttl=8760h -format=json \
  | jq -r '.auth.client_token' > /root/project1-token.txt
Enter fullscreen mode Exit fullscreen mode

The deploy script reads from /root/project1-token.txt. If you use GitHub Actions to automate deployments, store this token as a repository secret (VAULT_TOKEN_PROJECT1) and pass it to the fetch script. It can read project1's secrets and nothing else.

One limitation: reboots

When the VPS reboots, Vault starts in a sealed state. All requests are refused until you unseal it manually.

ssh root@YOUR_VPS_IP
VAULT_ADDR=https://127.0.0.1:8200 VAULT_SKIP_VERIFY=true vault operator unseal
# enter your unseal key
Enter fullscreen mode Exit fullscreen mode

Running containers keep their environment variables and continue working. But any new deployment would fail to fetch secrets until Vault is unsealed.

Auto-unsealing requires either a cloud-hosted HSM service or storing the unseal key on disk, which defeats the purpose. Manual unsealing after reboots is the right choice for a single-node hobby setup. Hetzner servers do not reboot unless you tell them to.

Back up your unseal key. You cannot recover it once it is lost.

Part 6 — Two instances →

The full series

  1. Why we are building this
  2. The server
  3. Traefik and SSL
  4. The first Supabase instance
  5. Vault, you are here
  6. Two instances
  7. Security and the load test

Top comments (0)