Secrets that live in git - encrypted, no Vault, no sidecar

#devops #security #rust #gitops

This is part of a series on building Orca, a single-binary orchestrator for the gap between Coolify and Kubernetes.

Every orchestrator eventually has to answer: where do the secrets go?

The default industry answer is "run a secret manager" - Vault, or a cloud KMS, or Infisical. And for a while, that's what Orca did too. It shipped a managed Infisical sidecar. Then one day I sat down to draw the dependency graph and realized the whole thing was eating its own tail.

The circular dependency nobody mentions

Think about it. Your orchestrator deploys your services. Your services need secrets. So the orchestrator runs a secret manager to hold them. But the secret manager is itself a service - it needs to be deployed, it has its own config, its own credentials, its own availability requirements. Who deploys the thing that holds the secrets for the things you deploy?

You've created a bootstrap problem and an availability coupling in one move. If the secret manager is down, deploys fail. If it's slow, deploys are slow. And you've added a stateful service to operate specifically so you could avoid operating stateful services carefully.

For a tool whose whole pitch is "no hidden dependencies," that was untenable. So I ripped it out and replaced it with something that has no moving parts at all.

Secrets as an encrypted file in your repo

The GitOps world already solved this, and the answer is SOPS + age. Here's the shape of it in Orca:

Secrets live in a single SOPS-format JSON file committed to your config repo, right next to your service.tomls.
Keys stay in plaintext; only values are encrypted. So a git diff on a secret change shows you which secret changed - just not its value. That readability is the entire reason to prefer SOPS over an opaque encrypted blob.
Values are encrypted to age recipients - the master's key and your personal offline key. Multi-recipient means you can always decrypt the file locally for recovery, completely independent of whether the master is up.
The master decrypts in-process at load. No sops or age binary is invoked at runtime; it's a Rust library call.

Configuring it is one block:

[secrets]
encrypted_file = "secrets.enc.json"            # lives in the config repo
age_key_file   = "/root/.config/orca/age.key"  # master identity, kept OUT of git
age_recipients = ["age1master…", "age1operator…"]

And using a secret is the same ${secrets.KEY} reference it always was - nothing about the service config changed.

The properties that fall out of this

This design isn't just "fewer services." It buys real properties:

Recovery is trivial. The disaster-recovery story for a secret manager is a runbook. Here it's a sentence: have the git repo and the age key. You can sops -d secrets.enc.json on your laptop and read everything. Orca doesn't need to be running, or even to exist, for you to get your secrets back.

Diffs stay honest. Because keys are plaintext, code review works. A PR that rotates db_password shows db_password changed. A PR that adds a new secret shows the new key. You review secret changes the same way you review config changes.

It's GitOps-native. The secrets file is just another file the reconciler converges on. When you orca secrets set, the master re-encrypts the file and commits + pushes it, so the repo stays the source of truth. (That write-back had a subtlety - if the master mutates the file but doesn't push, a later git pull could silently revert a secret. So the mutation path commits and pushes, with a pull --rebase retry. Getting that loop right mattered more than the crypto.)

Project scoping for free. Store a secret as myproject.db_url and it scopes to that project - a service in myproject resolves ${secrets.db_url} to the scoped value first, falling back to a global one. One flat file, prefix keys, no new machinery.

The honest caveats

Two things I'd want a reader to know before adopting this:

The in-process decryption relies on a pure-Rust SOPS implementation, which is young. I pin the version and round-trip test against files written by the real sops CLI, precisely because "trust me it's compatible" isn't good enough for the thing holding your database passwords.
"Deleted" secrets are only gone from HEAD. They're still in git history, decryptable by anyone with a recipient key. This is hygiene, not cryptographic erasure - true erasure means history rewrite and key rotation. Say what a thing does and doesn't do.

The lesson I took from all this: the best dependency is the one you delete. A secret manager felt like the professional choice. But the professional choice was recognizing that an orchestrator holding its own secrets, in the repo it already reconciles, with tools that work without it, is simpler and stronger.

Next up: the bug that taught me the most this year - a WebSocket session that was dead for ten days while my dashboard swore the node was healthy.