Coauthored with Ben de Haan
In a previous blog we talked about secure deployment. Secrets management is an important part of that. So what does that mean? In this blog we’ll give some pointers on how to do secrets management well in the perspective of a secure deployment. It’s easy to start saying “use tool X to store the secret” or “have all these detection tools in place!”, but that would lead to blind spots. Instead, let’s take a look at some pointers that would help you increase secret security holistically.
1. Inventory your secrets
Many security incidents actually stem from the fact that a “secret” was not treated as a secret, or was not properly protected. In addition to data leaks, there are other risks related to secrets: your application can get compromised when secrets are leaked. Many applications no longer function without their secrets. If you want to protect your secret well, you first need to know what you classify as a secret and where you store it currently. Based on that, you can take actions to further secure the secret.
2. Prepare to migrate them
Your technology stack evolves continuously, and you will switch eventually. You may have done a migration from “Jenkins” to “GitHub actions” or from your “secret shell script” to “ vault”. Similarly you can switch locations for your secret: from your Kubernetes Secrets to your “vault” or to your cloud provider (or vice a versa). All with good reason. The key aspect remains: you need to prepare to migrate a secret, because you want to make sure your secrets remain secret. Therefore you need to be able to tell what the purpose of the secret is and how it is consumed. It helps to store the metadata on a secret close enough to the secret so you can easily identify what/when you need to migrate.
3. Make secrets temporal
No secret is safe infinitely. The chances of a secret leak having major impact on your organization are a lot less if you tune down the validity. This means that you should make sure secrets have a limited lifetime and can be changed when needed. Depending on the type of secret it becomes easier to make them temporal. A cloud-provider access token, for instance, can be very short-lived. This is especially important for high-privileged access tokens. On the other hand: your TLS private keys might have to be stable a little longer in case you have to support a setup without perfect forward secrecy. If you can ensure that your public-private key pair only lives for a year, it is still “temporal”. Similarly, rotate your passwords of accounts you hold dear on a regular basis – a password manager makes this a whole lot easier.
4. Make sure they are not in code/config
Secrets in code/config are easy to read by others, and more importantly, easy to exfiltrate by tools. By now, it’s common knowledge that a secret should not be checked into git. However, places like your Terraform state bucket, environment configuration files, and your Kubernetes config maps, are often overlooked. In the ideal scenario your secrets only live in memory or, preferably, in software/hardware security modules. So when you generate your secret, a fair share of (secure) randomness should be involved. Your configuration code can entail length of a secret, the type, possible used alphabet, or other criteria, but it should not contain the secret itself. In this case, the instructions on how generating and using the secret are in code, but the actual secrets only exist in the software or hardware security module.
5. Bring the secret close to the intended recipient
If you expose a secret to more than just the intended recipient, you increase the risk of disclosing it to unauthorized parties. For instance: If a specific application on your Kubernetes cluster requires a secret, it’s better to let the application retrieve it instead of the worker node. There’s a simple reason for this: if all pods can access the secret through the worker node, any of these pods can leak it. This doesn’t just apply to Kubernetes, all technology stacks face similar versions of this problem.
6. Ensure least privilege access
Least privilege access in secrets management goes two ways:
- You want to make sure all entities only have the privileges they need – possibly temporarily (see #3)
- The secret scopes only to provide access to what its user requires
In practice, this means you will want to create separate secrets for separate functions or entities, and prevent unconstrained access. Sane defaults and temporary privilege elevation are key here. You don’t want everyone to be root all the time, and unconstrained secret access is by extension the same. If one of your team members gets compromised, you’ll be really happy the attacker has limited access by default.
7. Don’t operate by yourself
When handling secrets, whether through automated generation, or through manual import into your system: make sure you cooperate. Changing a secret can have a lot of impact on the consumers of the secret. Therefore, it’s good to do this together -pair programming style-, to make sure that no errors are made. Pair programming helps spreading the knowledge of the process as well! Last, you can help each other by ensuring copies of an imported secret are destroyed in case of a manual import.
8. Do not reuse secrets
Do not reuse passwords for various services, do not reuse keys for various purposes. Every time you use a secret for a given entity, and that entity gets compromised, so is your secret. Therefore a single purpose per secret helps to reduce the impact in case of an exposed secret.
9. Monitor secret access
There are various ways to monitor access to your secrets. We’ll discuss ‘traditional’ audit logging and monitoring, canaries, and external services detecting leaks concerning your organization.
Audit logging
The oldest and generally most robust way of monitoring whether a secret is accessed or modified, is simply using audit logs. This can be via application access-logs, cloud audit monitoring, or system services such as AuditD. Audit logs are a fundamental control in establishing what has happened. Things can get tricky, though: an attacker could use a compromised (service) account to obtain a secret, in which case you’d just see the (service) account accessing the secret. This means you won’t catch such an attack if you’re not correlating across different log sources.
Canaries
Some data breach stories involve an external entity informing an organization that it might have been breached. This doesn’t have to be the case. Like canaries in coal mines were used to warn of gas, you can put canary tokens, users, or access keys in various places. The moment these are accessed or used, they alert you of possible mischief, giving you an early warning (which might be your only one). One service that you could get started with is canarytokens.org.
HaveIBeenPwned
Another way of seeing whether a part of your secrets have been leaked, is checking haveibeenpwned. This service can tell you, based on your email address, whether your data has showed up in a breach. Please note that, if your email address did not show up, it does not mean it never got breached. Another indicator you can use is the pwned passwords service. This can tell you if the password of your choice already has been leaked before.
Pastebin / dark web monitoring
Not a real solution, but hey: you can try to monitor the internet yourself as well, to see if your secret ends up somewhere ;-). It is often better to ask specialized third parties to do this for you. Be advised: the effectiveness of these parties might vary.
Pro-tip: it is useful to enable tooling to monitor your own repositories! Even various open source tools can already help greatly!
10. Make secrets part of your threat model
Secrets are a key asset, so naturally they are also the object of several possible threats. When you threat model, be sure to include them! Note that it can be hard to quantify “secret is leaked” in terms of likelihood or impact. However, if you identify a threat as “Admin password in our git repo is reused to delete our main account”, then it becomes a lot easier to see what the possible level of risk is.
Where to go from here
These were a lot of pointers to start with! What should you do now?
- Make sure your inventory of secrets is up to date: Once you know which secrets you have, you can determine what to do with them.
- Make sure you move ahead in risk-based fashion: conduct threatmodeling exercises in order to prioritize what to do with which secret.
- If you’re late to the party and have a lot of secrets lying around, don’t worry: chunk up the work in small pieces, and most importantly: start!
If you have any questions, remarks, or additions, don’t hesitate to reach out!
Further reading
OWASP SAMM on secret management
Secure deployment
Exercises/examples on how(not) to use secrets
Canarytokens
Have I been pwned?
Gitleaks, Trufflehog
Top comments (0)