Originally published at woitzik.dev
I assumed my homelab repo was clean. No one had ever flagged anything in review (there is no one else reviewing it), CI was green, and I generally try to use Vault and ExternalSecrets for anything sensitive.
Then I ran a full-history gitleaks detect against it. It found 12 distinct secrets committed in plaintext — including the OIDC private key that signs SSO tokens for half the cluster.
This is the scanning setup I put in place afterward, the baseline strategy that let me adopt secret scanning without getting blocked by my own history on every commit, and the remediation plan for the leaks themselves.
View the complete homelab infrastructure source on GitHub 🐙
What Gitleaks Found
gitleaks detect --no-banner -v
Twelve real findings, plus one already-hashed password (lower severity but still shouldn't be hand-committed) and one false positive in ROADMAP.md (documentation text that happened to match a generic API key pattern).
The real findings, by severity:
| File | Secret | Why It Matters |
|---|---|---|
kubernetes/apps/authelia/configmap.yml |
OIDC issuer private key | Signs SSO tokens for ArgoCD, Vault, Grafana — highest blast radius |
kubernetes/apps/garage/config.yml |
RPC secret + admin token | Storage backend for Velero/Loki/CNPG backups |
kubernetes/apps/garage/secrets.yml |
Admin token (duplicate) | Same secret committed twice in two files |
terraform/stacks/network/local_backend.hcl |
Garage S3 access key | This is the Terraform state backend's own credential |
kubernetes/system/postgres/cnpg-backup-secret.yml |
Garage S3 secret key | Used for WAL archiving |
kubernetes/apps/paperless/secrets.yml |
Postgres password + AI API token | |
kubernetes/apps/cloudflared/secrets.yml |
Cloudflare Tunnel token | |
kubernetes/apps/headscale/config.yml |
OIDC client secret | Must match Authelia's client config |
kubernetes/system/monitoring/loki.yml |
Minio/S3 password | |
kubernetes/apps/mikrodash/secrets.yml |
Dashboard password | Lowest priority — internal tool only |
None of these were exposed by a public repo (this one is private), but "private repo" is not a security control — it's a single permission setting away from being public, and anyone with read access to the repo (or its history, forever) has all of this regardless.
Why a Private Repo Doesn't Make This Fine
The honest reason these accumulated: early in the project, before Vault and ExternalSecrets were set up, every new service got a quick secrets.yml with the actual values inline, "just to get it working." Once Vault was running, new services went through it — but nobody went back and migrated the old ones. Each individually felt low-risk at the time. Twelve of them, four months later, is a real exposure if the repo's access list ever changes.
This is the same drift pattern as the Terraform-vs-RouterOS-firewall divergence I wrote about separately: each shortcut is locally reasonable, the accumulated state is not.
Setting Up Gitleaks Without Getting Blocked by History
The naive approach — turn on gitleaks protect in pre-commit and call it done — fails immediately. Every single future commit gets blocked by the 12 pre-existing leaks, because gitleaks scans the whole working tree, not just your diff. You'd have to fix all 12 before you could make any other commit, including the commit that adds the scanning.
The fix is a baseline file:
gitleaks detect --baseline-path .gitleaks-baseline.json --no-banner -v
A baseline is a snapshot of currently-known findings. Anything in the baseline is allowed to keep existing; anything new fails the hook. Generate it once:
gitleaks detect --report-format json --report-path .gitleaks-baseline.json
Commit that baseline file. From this point forward, gitleaks only blocks genuinely new secrets — exactly what you want when adopting scanning on a repo with history older than the scanning itself.
The Three-Layer Hook Setup
One scan layer is not enough — a single missed git commit --no-verify or a commit made from a machine without the hooks installed slips through. Three layers, increasing scope, decreasing frequency:
# .pre-commit-config.yaml
- id: gitleaks-staged
name: Gitleaks (staged changes)
description: >-
Blocks committing NEW secrets. Uses .gitleaks-baseline.json so the
12 pre-existing leaks don't block every commit until they're fully
remediated — only genuinely new secrets fail this.
entry: bash -c 'gitleaks protect --staged --baseline-path .gitleaks-baseline.json --no-banner -v'
language: system
always_run: true
pass_filenames: false
stages: [pre-commit]
- id: gitleaks-full-repo
name: Gitleaks (full history, pre-push only)
description: Re-scans the entire repo and history before any push, against the same baseline.
entry: bash -c 'gitleaks detect --baseline-path .gitleaks-baseline.json --no-banner -v'
language: system
always_run: true
pass_filenames: false
stages: [pre-push]
gitleaks protect --staged at commit time — fast, scans only what's staged, catches a secret before it ever enters history.
gitleaks detect at push time — re-scans the entire repo (slower, but only runs once per push, not once per commit). This catches anything that slipped past the first layer, for example a commit made with git commit --no-verify.
CI runs the same gitleaks detect command as a third, environment-independent layer — catches anything pushed from a machine that never had the hooks installed at all.
Allowlisting Real False Positives
The ROADMAP.md false positive needed an explicit allowlist entry, not a baseline bypass — baseline entries are meant for things you intend to fix, allowlist entries are for things that were never secrets in the first place:
# .gitleaks.toml
[extend]
useDefault = true
[allowlist]
description = "Known false positives"
regexes = [
# ROADMAP.md doc text listing which services use which OIDC client auth
# method — matches the generic-api-key pattern but is plain documentation,
# not a secret.
'''Proxmox/PBS/Grafana/Headscale use `client_secret_basic`''',
]
Be specific with allowlist regexes. A broad pattern here defeats the entire point of scanning — match the exact false-positive string, not a category of strings that happens to include it.
The Remediation Plan
Finding the leaks and remediating them are two different projects. Remediation means: rotate the actual credential (not just remove it from the file — the old value is still valid until rotated), and move the new value into Vault behind an ExternalSecret so it never gets hand-committed again.
The tricky part is ordering. Some of these credentials are dependencies of each other:
1. Garage RPC secret + admin token + S3 keys
↳ Everything else's backups depend on Garage being internally consistent.
Rotating the S3 key also invalidates Terraform's own state backend
credential (terraform/stacks/network/local_backend.hcl uses the same
key) — update both in the same pass or Terraform loses access to its
own state.
2. Authelia OIDC issuer private key
↳ Highest blast radius if left exposed (signs every SSO session).
After rotating, every service trusting the old key should be checked
for unexpected active sessions.
3. Everything else, any order
↳ Cloudflare Tunnel token (rotate in Cloudflare dashboard first, update
second — order matters for tokens with an external source of truth).
↳ Headscale OIDC client secret must be rotated in lockstep with
Authelia's matching client config — they're a pair.
A secret with downstream dependents must be rotated with the dependents in mind, not in isolation. Rotating Garage's S3 key without immediately updating the Terraform backend config doesn't remove a vulnerability — it breaks Terraform's access to its own state.
Confirming Remediation Actually Worked
After moving a secret to Vault and rotating the credential, re-run the same scan:
gitleaks detect --baseline-path .gitleaks-baseline.json --no-banner -v
The secret will still show up — it's in history, and the baseline still lists it. That's expected; the baseline isn't meant to disappear until every listed item has actually been fixed. Only regenerate the baseline once all 12 are addressed, as a final confirmation step that nothing was missed in the process — not as a way to make individual items "go away" faster.
What This Doesn't Fix
Scanning catches secrets in files. It does not:
-
Scrub git history. The old values remain readable to anyone with repo access, forever, unless you rewrite history (
git filter-repo) — which has its own risks if anyone else has a clone. -
Replace rotation. A secret found and removed from the current file tree is still valid until you change the actual credential at its source (Cloudflare dashboard, Garage admin CLI, Postgres
ALTER USER, etc.). -
Catch secrets gitleaks' default ruleset doesn't recognize. Custom internal token formats need custom regex rules —
useDefault = truecovers known formats (AWS keys, generic API key patterns, JWTs) but not everything.
The same baseline-adoption pattern applies directly to any enterprise repo with years of history and no prior secret scanning — which describes most codebases that predate a security initiative. The Vault + ExternalSecrets target architecture this remediation moves toward is the same pattern covered in External Secrets Operator + HashiCorp Vault — that's where these 12 secrets are headed.
Top comments (0)