The de facto standard for carrying secrets on our servers has been the same for years: you open a .env file, write the password into it, add an EnvironmentFile= line to the unit file, and you are done. The greatest advantage of this method is that it works. Its greatest disadvantage is exactly the same: because it works, nobody questions it. Yet an environment variable is a configuration transport mechanism, not a secret storage mechanism — and systemd has offered a replacement for years.
The Crime Scene on My Own Server
What made me decide to write this article was not somebody else's code. I noticed it while looking at my own blog's deploy directory. mustafaerbay.service is hardened like this:
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/opt/mustafaerbay /var/lib/mustafaerbay /var/log/mustafaerbay
The file system is locked down, privilege escalation is off, /home is read-only. Then, a few lines above, this sits:
EnvironmentFile=-/etc/mustafaerbay.env
The same file is loaded by mustafaerbay-digest.service and pipeline-health.service as well; the SMTP credentials for the subscription newsletter and the alert mails live in there. So I had tightened the sandbox with the discipline I described in a separate article, while inheriting my secret transport from the 1990s. Rather like fitting a steel door and leaving the key under the doormat. Finding this combination in your own infrastructure is easier than you would think.
Why an Environment Variable Cannot Keep a Secret
The "the file mode is 600, nobody but root can read it" defence sounds convincing at first. The problem is not in the file, but in the variable's life after it leaves the file. systemd's own systemd.exec(5) documentation is unusually blunt here: environment variables are exposed to unprivileged clients over D-Bus IPC and are generally not understood as data that requires protection.
The second issue is more insidious: environment variables propagate down the process tree. Every child process your service spawns — an sh -c wrapper, a backup script, a helper tool invoked from a shell — inherits that block. The propagation crosses setuid/setgid boundaries too. You think you have given your database password only to the application; in practice you have given it to the entire lineage of processes that application will ever spawn. Add to that the possibility of it being read through /proc during a crash, or an error-reporting library dumping the environment verbatim into a log.
I have walked through the leak paths of environment variables before; that article was the diagnosis, this one is the treatment. Credentials target both of these problems directly. The secret does not propagate down the process tree, and an access check is enforced by the kernel every time it is accessed.
What a Credential Actually Is
The mechanism is simpler than you would expect: systemd places the secret as a file in a directory specific to that service, and announces the path of that directory through the $CREDENTIALS_DIRECTORY variable. The variable holds no secret, it holds the path to the secret — that difference is the summary of this whole article.
Four properties of this directory matter. It is read-only. Where possible it is kept in non-swappable memory, that is, on ramfs. It is accessible only to the user the unit is associated with via User= or DynamicUser= (and to root). And the file mode is 0400; the systemd-creds list command flags any credential that deviates from this as insecure.
Its interaction with DynamicUser= is particularly elegant: privileged data can reach the service even when the UID is not known in advance, because systemd reads the credential and produces a copy specific to the unit. The source file does not need to be readable by the service user at all.
The simple form looks like this:
[Service]
LoadCredential=db-password:/etc/credstore/db-password
ExecStart=/usr/bin/myapp --password-file ${CREDENTIALS_DIRECTORY}/db-password
If you do not specify a path, systemd looks in /etc/credstore/, /run/credstore/ and /usr/lib/credstore/ itself. In an Environment= line you can use the %d shorthand (Environment=PGPASSFILE=%d/db-password). If you need to embed a path in the configuration file of software that does not support credentials natively, system services also have the fixed path /run/credentials/UNIT_NAME — but the portable one is $CREDENTIALS_DIRECTORY, because it works for user services too.
One limit: the total credential size per unit is 1 MB. More than enough for passwords, keys and certificates; not meant for carrying data.
Encryption: Protecting the Secret at Rest Too
Everything so far protects the secret at runtime, while on disk it still sits in plaintext. This is where systemd-creds comes in. Encryption is done with AES256-GCM, so you get both confidentiality and integrity. The key can come from three sources:
-
host: the key in
/var/lib/systemd/credential.secret, accessible only to root. - tpm2: a key derived from the machine's TPM2 chip, never written to disk.
- host+tpm2: a combination of the two — decrypting requires access to both the chip and the OS installation.
If you do not pass --with-key=, the default is auto: the TPM2 key is used if a TPM2 device is found and you are not running in a container, and the host key is used if /var/lib/systemd/ is on persistent media. On a typical physical server that means the combination of both. The practical consequence: the encrypted credential you produce cannot be opened anywhere but that machine. An attacker trying to extract secrets from a leaked backup goes home empty-handed.
It takes two commands:
# Encrypt the file, then securely remove the plaintext
systemd-creds encrypt --name=db-password plaintext.txt /etc/credstore.encrypted/db-password
shred -u plaintext.txt
On the unit side, a single line changes — LoadCredentialEncrypted= instead of LoadCredential=. Decryption and authentication happen automatically at the moment of service activation; since the information about which key is required is embedded inside the encrypted data, you do not have to specify anything extra.
Embedding the encrypted blob directly into the unit file instead of a separate file is also possible:
systemd-ask-password -n | systemd-creds encrypt --name=mysql-password -p - -
This command prints a ready-to-paste SetCredentialEncrypted= line. Dropping it into /etc/systemd/system/xyz.service.d/50-password.conf and running daemon-reload is enough.
Watch out for one trap here: do not be misled by the similarity between SetCredential= and SetCredentialEncrypted=. The former takes the value in plaintext and, per the documentation's explicit warning, is visible to unprivileged processes over IPC — so it is not for secrets, only for non-sensitive data such as user names or public key material. When a secret is involved, use either LoadCredential* or SetCredentialEncrypted=.
The --name= parameter is not merely cosmetic. The name is embedded into the encrypted data and verified on decryption, so a credential minted as smtp-password cannot be reused by another unit as admin-token. You can also embed an expiry with --not-after=; once the date has passed, decryption fails. A quiet but useful door for granting temporary access.
Four Details That Will Hurt You in Production
The beauty of the mechanism has been described; now the bill.
Binding to TPM2 is an irreversible decision. If the motherboard dies, the TPM is cleared, or you replace the machine, those credentials will never open again. This is not a bug, it is the design itself — but it assumes your disaster recovery plan keeps the source of the secrets somewhere else. In other words, systemd credentials do not replace your secret distribution layer; they harden the final stop of secrets on the server. The source should still be a sops + age repository, a vault, or your password manager.
PCR binding breaks on updates. If you bind to specific PCR values with --tpm2-pcrs=, the measurements change after a firmware or kernel update and the secret will not open. It is no accident that the default is to bind to no PCRs at all. If you want a binding that survives updates, use the signed PCR policy via --tpm2-public-key-pcrs=: this binds not to specific values but to any set of values for which a signature can be provided, and its default is PCR 11. A continuation of the logic in the article on Secure Boot and TPM as a root of trust.
Rotation requires a restart. A credential is only read and decrypted during service activation. Writing the new password into the credstore is not enough; you have to restart the unit. If you want zero-downtime rotation, design it together with the application's reload capability; otherwise your rotation window is your restart window.
Failure behaviour differs per directive, and teams unaware of this difference experience silent faults. If a credential defined with LoadCredentialEncrypted= or SetCredentialEncrypted= cannot be decrypted, the service does not start — a loud fault that is easy to diagnose. But if a credential picked up via ImportCredential= cannot be decrypted, only a warning is generated, the credential is skipped silently, and the service keeps running. Your application comes up with a missing secret and you do not find out until the first request. The documentation offers the remedy too: ConditionCredential= skips the unit silently when the credential is absent, while AssertCredential= fails loudly. Pick one of them deliberately instead of running on quietly with a hole.
The shared bill for these four items arrives when you go fleet-wide. Because ciphertext produced with the auto default is machine-specific, you cannot hand the same SMTP password to fifty servers by copying one file: either you build a configuration-management step that re-encrypts on every host, or you move to offline sealing with the --tpm2-public-key= option mentioned above. In containers, TPM2 is not in play at all — auto skips TPM2 inside a container, so for services you run under Podman Quadlet the protection effectively falls back to the host key. Design for that from the start.
How the Secret Reaches the Machine
Everything so far applies once the secret is already on the server. But how did it get there? systemd covers this question too: the service manager itself can receive "system credentials" and distribute them to units.
On virtual machines the SMBIOS OEM string table is used. On the QEMU side there are two alternative routes; you can pass the same secret through either:
# Preferred: SMBIOS type 11
qemu-system-x86_64 -smbios type=11,value=io.systemd.credential:mycred=supersecret
# Alternative: qemu fw_cfg
qemu-system-x86_64 -fw_cfg name=opt/io.systemd.credentials/mycred,string=supersecret
The documentation favours SMBIOS between the two: it is faster and less specific to the chosen VMM implementation. There is also a 55-character limit on names passed via fw_cfg, so settings with long names may not fit. If you want to carry binary data, you pass a Base64-encoded value with the io.systemd.credential.binary: prefix. Container managers, the initrd, the UEFI environment via systemd-stub, and cloud providers' instance metadata services (IMDS) all feed the same pool. The credential is injected into the machine at first boot, and services pick it up with ImportCredential=.
ImportCredential= accepts globs, but with a restricted grammar: only a single trailing * is allowed, and ? and [] are not supported. Renaming with a colon is possible too — writing ImportCredential=my.original.*:my.renamed. makes every credential starting with my.original. available to the service as my.renamed.xxx. Practical for aligning the name the application expects with the name the provider gives.
So far the secret's source has been either a file on disk or a value embedded in the unit file. If you need dynamic delivery, there is a third path: LoadCredential= can point to an AF_UNIX stream socket instead of an absolute path. systemd connects to it while starting the process and reads the secret. The other side can see which unit is requesting which credential via getpeername(2), because the socket name contains the requesting unit and the credential ID. A natural integration point for anyone wanting to serve many consumers from a single secret distribution service.
How to Verify That It Works
The most dangerous state in secret management is a setup you believe is working. Two commands remove that uncertainty.
The first is testing the credential by decrypting it inside a transient service, without changing anything permanent:
systemd-run -P --wait \
-p LoadCredentialEncrypted=db-password:/etc/credstore.encrypted/db-password \
systemd-creds cat db-password
If the encrypted file really does open on this machine, you will see the password on screen; if it does not, you have found out before taking down the production service.
The second is systemd-creds list from within the service context. It dumps each credential's name, size and security state: secure (in non-swappable memory), weak (in another kind of memory) or insecure (mode other than 0400, meaning others besides the owner can read it). Those three words are the shortest report telling you whether your setup actually delivers the protection it claims.
Version Gap: v261 Today, v262 Tomorrow
This area is still moving, so which systemd you are on matters. At the time of writing, the latest stable release is v261; v262 is still at the release-candidate stage (v262-rc1, 1 September 2026) and brings two notable changes on the credential side.
First, TPM-sealed credentials are now pinned to the TPM's SRK. The purpose is to protect communication with the TPM against an interposing attacker stealing the decrypted secret. The compatibility note is critical: TPM-bound credentials minted after this change are not recognised by older systemd versions — while systemd continues to accept credentials created before the change. In a mixed-version fleet, if your credential minting machine is on the newest version, your secrets will not open on the older servers.
Second, the systemd.credentials_boot_policy= kernel command line option is arriving. It controls when boot credentials encrypted with the "null" key — which provide neither confidentiality nor authentication — are accepted: strict (never), tofu (first boot or no TPM2), relaxed (SecureBoot disabled or no TPM2 — the default and the previous behaviour) and off (always). It does not affect credentials encrypted with host or TPM2 keys. If you use null-key credentials while preparing server images, you will need to choose this policy deliberately before moving to v262.
Where to Start
Do not migrate everything overnight. In my own infrastructure I find this order reasonable: inventory first — run grep -rl "EnvironmentFile\|Environment=.*\(PASS\|TOKEN\|KEY\|SECRET\)" /etc/systemd/system/ /usr/lib/systemd/system/ to map the crime scene. Then pick the single unit with the highest impact; in my case that is /etc/mustafaerbay.env, which spreads SMTP credentials across three separate units. Encrypt the secret with systemd-creds encrypt --name=, place it under /etc/credstore.encrypted/, use LoadCredentialEncrypted= in the unit, and hand the password to the application as a file path.
If the application insists on an environment variable, you have not lost: in an ExecStart wrapper, read the variable from the credential file and give it only to that process. The secret no longer sits on disk in plaintext and no longer spreads across the whole process tree — that is where most of the gain lies.
Finally, note this: there is also a systemd.set_credential= kernel command line option, but it is not recommended for secrets because unprivileged userspace can read the kernel command line. The documentation is honest enough to warn about itself; let us carry that honesty into our unit files.
Environment variables were never designed to hold secrets. The reason we keep using them for that job is not safety, but habit. And the cost of changing the habit is just what I described here — a few lines of unit file and one encrypt command.
Top comments (0)