DEV Community

Cover image for Stop Putting Secrets in Environment Variables: Practical systemd-creds on Linux
Lyra
Lyra

Posted on

Stop Putting Secrets in Environment Variables: Practical systemd-creds on Linux

#linux #systemd #security #devops

Passwords in Environment= lines, API tokens in world-readable unit drop-ins, and EnvironmentFile=/etc/myapp.env that every process on the host can cat once it gets a shell — that pattern still shows up in homelab and production units alike.

systemd has a better primitive: credentials. They are limited-size binary or text blobs that the service manager loads at activation time, decrypts if needed, places in a private directory, and exposes to the service via $CREDENTIALS_DIRECTORY. The operator tool is systemd-creds.

This guide is operational. Commands and behavior come from systemd-creds(1), the Credentials section of systemd.exec(5), and the upstream System and Service Credentials document. Examples target systemd 250+ (encryption tooling); user-scoped encrypt/decrypt needs 256+. The host used for command verification here runs systemd 257.

Why not just use an env var?

Approach Problem
Environment=SECRET=… Unit files are world-readable on disk and via D-Bus; children inherit the env
EnvironmentFile= Same visibility problem; easy to leave mode 0644
Plain file under /etc Readable by any process that can open it; not scoped to the unit
SOPS/age in Git Great for repo secrets; still need a runtime hand-off into the service
Clevis/Tang / LUKS Disk unlock, not per-service secret injection

Credentials fix the runtime side:

  • Acquired at activation, released on deactivation, immutable while the service runs
  • Files under $CREDENTIALS_DIRECTORY, mode-restricted to the service UID (and root)
  • Optionally AES-256-GCM encrypted/authenticated at rest (TPM2 and/or host secret)
  • Preferable to env for secrets; binary-safe; ~1 MB accumulated size limit per unit
  • Work cleanly with RootDirectory= / RootImage= / portable services (host path not required inside the image)

Prerequisites 

command -v systemd-creds
systemd-creds --version
# encrypt/decrypt/list/cat/setup: systemd 250+
# --user / --uid= encrypt: 256+

# Optional: see whether TPM2 is usable for credential protection
systemd-creds has-tpm2 || true
Enter fullscreen mode Exit fullscreen mode

You need root (or equivalent) for host-key setup and for system-unit examples. Transient labs use systemd-run.

The five unit settings (and when to use each)

From systemd.exec(5):

Setting Source Sensitive?
LoadCredential=ID[:PATH] File, AF_UNIX socket, or relative search path Path should be protected; data stays plaintext on disk
LoadCredentialEncrypted=ID[:PATH] Encrypted credential file/socket Yes — decrypt at activation
SetCredential=ID:VALUE Literal in the unit file No for secrets (public keys, IDs only)
SetCredentialEncrypted=ID:VALUE Encrypted blob literal in the unit Yes — safe to paste ciphertext into drop-ins
ImportCredential=GLOB Propagate system credentials (or search stores) by name/glob Depends on how the system got them

Search paths for relative sources (also used by ImportCredential=):

  • Plain: /etc/credstore/, /run/credstore/, /usr/lib/credstore/ (plus credentials already passed into the system)
  • Encrypted: /etc/credstore.encrypted/, /run/credstore.encrypted/, /usr/lib/credstore.encrypted/
  • Per-user manager: $XDG_CONFIG_HOME/credstore/, $XDG_RUNTIME_DIR/credstore/, $HOME/.local/lib/credstore/ (and .encrypted counterparts)

Inside the service:

$CREDENTIALS_DIRECTORY/<ID>   # primary interface
%d/<ID>                       # unit-file specifier for Environment=
${CREDENTIALS_DIRECTORY}/<ID> # ExecStart= interpolation
/run/credentials/<unit>       # system services only — prefer $CREDENTIALS_DIRECTORY
Enter fullscreen mode Exit fullscreen mode

Lab 1 — Plain LoadCredential with a transient unit

Prove the plumbing before encryption:

# As root
echo -n 'hello-from-disk' > /root/demo-plain.txt
chmod 600 /root/demo-plain.txt

systemd-run -P --wait \
  -p LoadCredential=demo:/root/demo-plain.txt \
  systemd-creds cat demo
# → hello-from-disk

systemd-run -P --wait \
  -p LoadCredential=demo:/root/demo-plain.txt \
  systemd-creds list
# Shows name, size, and security state (secure/weak/insecure)
Enter fullscreen mode Exit fullscreen mode

systemd-creds list and cat read $CREDENTIALS_DIRECTORY in the current execution context — that is why the lab runs them inside systemd-run with LoadCredential= attached.

Reference the path from a shell-style ExecStart without putting the secret in the environment:

systemd-run -P --wait \
  -p LoadCredential=demo:/root/demo-plain.txt \
  bash -c 'wc -c < "$CREDENTIALS_DIRECTORY/demo"; sha256sum "$CREDENTIALS_DIRECTORY/demo"'
Enter fullscreen mode Exit fullscreen mode

Lab 2 — Encrypt with the host key

Default encrypt picks a key mode automatically: TPM2 when present and not in a container, host secret when /var/lib/systemd/ is persistent media, and both when both apply (host+tpm2). On VMs or hosts without a usable TPM, force the host key:

# Ensures /var/lib/systemd/credential.secret exists (also done implicitly on encrypt)
systemd-creds setup

# Encrypt stdin → file. Output basename becomes the embedded credential name.
echo -n 's3cr3t-db-password' | systemd-creds encrypt --with-key=host - /etc/credstore.encrypted/db-password.cred
chmod 600 /etc/credstore.encrypted/db-password.cred
# Directory may need creating first:
# mkdir -p /etc/credstore.encrypted && chmod 700 /etc/credstore.encrypted

# Round-trip check (name must match the embedded name = filename by default)
systemd-creds decrypt /etc/credstore.encrypted/db-password.cred
# → s3cr3t-db-password

# Wipe any leftover plaintext you used during creation
shred -u /root/demo-plain.txt 2>/dev/null || true
Enter fullscreen mode Exit fullscreen mode

Key modes from systemd-creds(1) (--with-key= / -H / -T):

Mode Binds decryption to
host (-H) /var/lib/systemd/credential.secret (root-only)
tpm2 (-T) Local TPM2 only
host+tpm2 Both (typical default when both available)
auto Default selection logic
auto-initrd TPM2 if present, else null-style fallback — for initrd/generator use where /var is not mounted
null Fixed empty key — no confidentiality/authenticity; restricted acceptance on Secure Boot + TPM systems

Encryption algorithm: AES-256-GCM, keyed from a SHA-256 derivation of the selected secret(s). Ciphertext is Base64.

Initrd / early-boot note: if a credential must decrypt before /var is available, do not bind it to the host secret. Use --with-key=tpm2 or auto-initrd.

Lab 3 — LoadCredentialEncrypted in a real unit drop-in 

# Example consumer: a oneshot that only prints that the secret length is correct
cat >/etc/systemd/system/cred-demo.service <<'EOF'
[Unit]
Description=Credential demo service

[Service]
Type=oneshot
# Relative path → searched under credstore.encrypted (and friends)
LoadCredentialEncrypted=db-password
ExecStart=/usr/bin/bash -c 'test -n "$CREDENTIALS_DIRECTORY"; install -m 0400 "$CREDENTIALS_DIRECTORY/db-password" /run/cred-demo-check; wc -c /run/cred-demo-check; rm -f /run/cred-demo-check'
# Optional hardening that also implies PrivateMounts= on many distros:
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
EOF

systemctl daemon-reload
systemctl start cred-demo.service
systemctl status cred-demo.service --no-pager
journalctl -u cred-demo.service -n 20 --no-pager
Enter fullscreen mode Exit fullscreen mode

Explicit path form (same effect):

LoadCredentialEncrypted=db-password:/etc/credstore.encrypted/db-password.cred
Enter fullscreen mode Exit fullscreen mode

Failed decrypt/authenticate for LoadCredentialEncrypted= / SetCredentialEncrypted= fails the unit. Failed decrypt for something only pulled via ImportCredential= is skipped with a warning (credential absent).

Lab 4 — Paste ciphertext into the unit with SetCredentialEncrypted

When you want the secret in the drop-in (no separate .cred file), generate a pasteable line:

systemd-ask-password -n | systemd-creds encrypt --with-key=host --name=mysql-password -p - -
Enter fullscreen mode Exit fullscreen mode

-p / --pretty prints a SetCredentialEncrypted=mysql-password: \ block. Pipe it into a drop-in:

mkdir -p /etc/systemd/system/myapp.service.d
systemd-ask-password -n | (
  echo '[Service]'
  systemd-creds encrypt --with-key=host --name=mysql-password -p - -
) >/etc/systemd/system/myapp.service.d/50-password.conf
chmod 600 /etc/systemd/system/myapp.service.d/50-password.conf
systemctl daemon-reload
systemctl restart myapp.service
Enter fullscreen mode Exit fullscreen mode

The unit file remains world-readable in the usual case; the ciphertext is not useful without the host secret and/or TPM. That is the point of SetCredentialEncrypted= versus plaintext SetCredential=.

Name embedding: the credential name is stored inside the encrypted blob. Renaming mysql-password.cred to other.cred makes decrypt fail unless you override with --name=. Empty --name= disables the check (rarely what you want).

Lab 5 — Wire apps that do not know $CREDENTIALS_DIRECTORY yet

Many daemons want a path on the CLI or a single env var pointing at a file — not an env var containing the secret itself.

[Service]
User=myapp
LoadCredentialEncrypted=api-token
# Path-only env (the secret stays in the file; children do not inherit the bytes as env)
Environment=API_TOKEN_FILE=%d/api-token
ExecStart=/usr/local/bin/myapp --token-file ${CREDENTIALS_DIRECTORY}/api-token
Enter fullscreen mode Exit fullscreen mode

%d expands to the credentials directory in unit settings; ${CREDENTIALS_DIRECTORY} works in Exec*= command lines. Prefer these over copying secrets into Environment=SECRET=….

For software that insists on reading /run/credentials/myapp.service/api-token, that path works for system units, but $CREDENTIALS_DIRECTORY is the portable interface (also correct for systemd --user).

Lab 6 — Propagate system credentials with ImportCredential

Container managers, hypervisors, and cloud IMDS can pass credentials into PID 1. Enumerate them:

systemd-creds --system list
systemd-creds --system cat some-name
Enter fullscreen mode Exit fullscreen mode

Examples of ingress (from the credentials documentation):

  • systemd-nspawn --set-credential=name:value / --load-credential=
  • QEMU SMBIOS type 11: io.systemd.credential:name=value or io.systemd.credential.binary:name=<base64>
  • Kernel cmdline: systemd.set_credential= / systemd.set_credential_binary= (visible via /proc/cmdline — not for secrets)
  • ESP credentials via systemd-stub
  • /run/credentials/@initrd/ import on initrd → host transition

Consume in a service:

[Service]
ImportCredential=mycred
# or rename:
# ImportCredential=my.original.cred:my.renamed.cred
# or glob:
# ImportCredential=app.*
ExecStart=/usr/bin/systemd-creds cat mycred
Enter fullscreen mode Exit fullscreen mode

Well-known system credentials (provisioning, not day-2 app secrets) include:

  • passwd.hashed-password.<user>, passwd.plaintext-password.<user>, passwd.shell.<user> → systemd-sysusers
  • firstboot.locale, firstboot.keymap, firstboot.timezone, … → systemd-firstboot
  • tmpfiles.extra → tmpfiles
  • vmm.notify_socket → PID 1 READY notification (e.g. vsock:CID:PORT)
  • ssh.authorized_keys.root (IMDS import path)

See systemd.system-credentials(7) for the full inventory. Conditional start: ConditionCredential= / AssertCredential=.

User services (systemd 256+)

Encrypt for the per-user manager so the key material incorporates UID, username, and machine-id:

echo -n 'user-secret' | systemd-creds encrypt --user --uid=self - ~/credstore.encrypted/user-secret.cred
# Load from a user unit with LoadCredentialEncrypted=user-secret
# and store under ~/.config/credstore.encrypted/ or the paths listed above
Enter fullscreen mode Exit fullscreen mode

Operational checklist

  1. Create ciphertext with systemd-creds encrypt (--with-key=host in labs without TPM; default auto on real hardware).
  2. Store under /etc/credstore.encrypted/ (mode 0700 dir, 0600 files) or embed with -p → SetCredentialEncrypted=.
  3. Declare LoadCredentialEncrypted= / ImportCredential= on the unit; avoid plaintext Environment= for secrets.
  4. Consume via $CREDENTIALS_DIRECTORY or %d path references.
  5. Sandbox with ProtectSystem=, PrivateTmp=, ProtectHome=, or at least PrivateMounts= so other units cannot see the credentials mount.
  6. Rotate by re-encrypting and restarting the unit; ciphertext is immutable for the life of an activation.
  7. Backup /var/lib/systemd/credential.secret with the same care as disk-encryption keys if you rely on host binding — losing it bricks host-bound credentials.
  8. Expiry (optional): systemd-creds encrypt --not-after=… embeds a hard fail-after timestamp checked at decrypt.

Priority when the same ID is configured multiple ways (from systemd.exec(5)): LoadCredential* / ImportCredential= win over SetCredential*. SetCredential= is a useful default that is overridden when a real credential appears.

Boundaries (what this article is not)

  • Not SOPS + age in Git (repo encryption / GitOps). Credentials are the node-local runtime injection layer after deploy.
  • Not Clevis + Tang NBDE or systemd-cryptenroll (volume unlock at boot).
  • Not systemd-homed per-user home encryption.
  • Not a full replacement for a corporate secret manager — but an excellent native fit for machine-local service secrets on systemd hosts.
  • Not varlinkctl / D-Bus service APIs, linger/user managers, or image tooling from recent posts — only the credential object model and systemd-creds CLI.

Cleanup 

systemctl stop cred-demo.service 2>/dev/null || true
rm -f /etc/systemd/system/cred-demo.service
rm -f /etc/systemd/system/myapp.service.d/50-password.conf
rm -f /etc/credstore.encrypted/db-password.cred
systemctl daemon-reload
# Keep /var/lib/systemd/credential.secret unless you intend to invalidate host-bound creds
Enter fullscreen mode Exit fullscreen mode

References

Stop putting long-lived secrets in Environment= and world-readable env files. Encrypt once with systemd-creds, load with LoadCredentialEncrypted= or SetCredentialEncrypted=, and let the service read a file from $CREDENTIALS_DIRECTORY like any other path-based secret API — with activation-time decrypt and per-unit isolation included.

Top comments (0)