For a long time my homelab's secrets lived the way most people's do: a database
password in one docker-compose.yml, an API key in another, a token pasted into
a .env that got copied to three machines and diverged on all of them. Nothing
was wrong, exactly. It was just that no single place could answer the question
"where does this credential live, and what still uses it?" — and that question
turns out to be the whole game.
So I moved everything behind a self-hosted secrets manager
(Infisical, though the shape here applies to Vault,
OpenBao, or any of them). One rule, ruthlessly applied:
Every secret has exactly one home and one system of record. Nothing keeps its
own copy on disk.
This is what that actually looks like in practice, what it fixed, and the three
traps that cost me real time — including the one where the tool that stores my
secrets couldn't hand one back because of a single $.
The principle: one home, pulled at runtime
Before: a secret was wherever I'd last needed it — a compose file, a config, a
shell history, a backup of all three. Rotating one meant hunting through the
estate hoping I'd find every copy, and knowing I couldn't be sure I had.
After: a service holds no credentials on disk. It boots, authenticates to the
secrets manager with a machine identity, pulls exactly the secrets it needs
into memory, and runs. The values exist on disk in exactly one system, encrypted,
and nowhere else.
The test I use to know it's working is brutal and simple: can I rotate any
password in the estate by changing it in exactly one place? When I audited mine,
the best thing I could say about the whole setup was yes — because not one
docker-compose.yml held a real secret to go stale. That's the entire payoff, and
it's worth the setup cost on its own.
Machine identities, not humans
The thing I'd most want a new self-hoster to internalise: the things pulling
secrets are not you. Each service authenticates as itself — a machine identity
scoped to only the folder of secrets it needs. The monitoring stack's identity can
read the monitoring secrets and nothing else. The AI agent's identity reads its
keys and nothing else.
Two consequences fall straight out of that:
- Blast radius is scoped by default. A compromised service can read its own secrets, which it already had in memory anyway — not the whole vault. The identity is the boundary.
- Access is auditable and revocable per-service. Retiring a service means revoking one identity, not wondering which shared token it might have known.
And one rule I now treat as non-negotiable, because an autonomous agent runs
against this vault: the automated identity gets read-only. The agent can
read the secret it needs to do its job. It cannot write, rotate, or delete one.
Writing secrets is a human action, gated behind a human identity. A clever process
with write access to the place all your credentials live is a very bad day waiting
for a bug.
The resolver: degrade in a known direction
Services don't call the vault's API directly. They go through a tiny resolver with
a fixed precedence:
environment variable → secrets manager → built-in default
That order is deliberate, and each step earns its place:
- Environment first lets me override anything for a one-off — a debug run, a test container — without touching the vault.
- Secrets manager second is the real source of truth for everything normal.
- Default last means a non-secret config value still has a sane fallback if the manager is briefly unreachable — but a genuine secret's "default" is deliberately a value that fails closed, not a working credential. If the vault is down, the service fails to authenticate loudly; it does not silently fall back to some baked-in key. Degrading toward "broken and obvious" beats degrading toward "working with a secret I forgot was hardcoded."
The resolver is thirty lines. It's the single most-used piece of code in the
estate, precisely because it's boring.
Trap 1: the bootstrap secret — who guards the guard?
The obvious hole in "no secret on disk" is the machine-identity credential itself.
That has to live somewhere a service can read at boot, before it can talk to the
vault. You cannot store the key to the vault inside the vault.
My answer isn't clever, it's just honest about the trade: the bootstrap
credential lives in one tightly-permissioned environment file per host, owned by
root, 600, and it is the only secret on that box. Everything else derives
from it at runtime. I've accepted that there is exactly one secret I can't
centralise, so I make sure there's only one, I know exactly where it is, and it's
the first thing I'd rotate if a host were ever touched.
The lesson generalises: a secrets manager doesn't eliminate the on-disk secret,
it collapses N of them to one. Pretending it's zero is how you end up not
guarding the one that matters most.
Trap 2: a single $ broke the whole stack
This one cost me an evening. A service read its secrets from an env_file in
compose, and it restart-looped on boot with an authentication failure — using a
password I could see, plainly correct, sitting right there in the file.
The password contained a $. Docker Compose performs variable interpolation
on values in an env_file, so pa$$word became pa followed by two empty
variable expansions. The service received a truncated password and, of course,
the failure didn't say "your password was mangled by interpolation" — it said
"authentication failed," which sent me hunting in entirely the wrong place.
The fix is to escape the $ (double it: $$) or, better, avoid $ in generated
secrets entirely. But the lesson is the one that keeps recurring across this
whole series: a layer between you and your secret will transform it, and the
error will blame the wrong component. When a credential you can literally see is
"wrong," suspect the pipe it travelled through before you suspect the value.
Trap 3: the wrong master password, silently
I keep a tiered rotation worklist — the important credentials, staged to be
rotated on a schedule. The rotation job stalled, and the reason was almost funny:
the master password for the vault itself, stored so the automation could log
in, was wrong. Not expired — wrong, a stale copy from before a change.
So the machinery designed to keep every other secret fresh was blocked by a
stale copy of the one secret it needed to run. The secrets manager was fine; my
record of how to get into it wasn't. It's the bootstrap problem from Trap 1
wearing a different hat: the credential that unlocks the system is the one most
likely to be quietly out of date, because it's the one thing the system can't
manage on your behalf.
The habit that ties it together: fingerprint, never print
Running all this taught me one operational reflex I now apply everywhere:
verify a secret by its fingerprint, never by printing its value.
When I need to confirm that the copy in the config, the copy in the vault, and the
copy in the running container all agree, I compare their SHA-256 hashes. I
never echo the secret to a terminal — because the instant you do, it's in a
scrollback buffer, a shell history, and possibly a terminal-recording tool, and
you've quietly created a new, un-managed copy of the exact thing your entire setup
exists to keep in one place. You can prove two secrets are identical without either
of you ever seeing the value. Do that.
What I'd tell you to copy
- One home per secret, pulled at runtime. The win isn't encryption — it's being able to rotate anything in one place and know nothing stale survives.
- Machine identities, scoped narrow. The identity is the boundary. A service reads its own folder and nothing else.
- Automated identities get read-only. Reading a secret is a job. Writing one is a human decision.
- Resolve through a tiny layer that fails closed. Env → vault → a default that's safe to be missing, never a real credential hiding as a fallback.
- Admit there's one secret you can't centralise, and guard that one. The bootstrap credential and the vault's own master password are the two most likely to be stale, because they're the two the system can't manage for you.
- Fingerprint, never print.
None of this made my homelab more impressive to look at. It made it legible —
I can now answer "where does this live and what uses it?" for every credential in
the place, which is the question I couldn't answer before, and the only one that
matters when something leaks.
I wrote the longer version up as a field report — the full resolver pattern, the
bootstrap trade-off, all three traps, and a checklist you can run against your own
estate. It's
*pay-what-you-want, including free*.
Take the checklist, pay nothing if you like.
Field notes from moving a sprawl of scattered .env files into one self-hosted
secrets manager, and breaking it in most of the ways available. Names, paths and
values here are illustrative — the point is the shape, not my folder layout.
🤖 Drafted with AI assistance from my own homelab notes, logs and repos, then reviewed and edited before publishing.
Top comments (0)