Recently I faced the decision whether to use a secrets manager or not.
Secrets managers provide some convenient functionalities to deal with secrets, but they come at the cost of another dependency, add a point of failure, complexity and increase provider lock-in.
I think if you are unsure about whether or not to use a secrets manager, you probably don't need one...
A common misconception about secret managers
It took me some time to understand the need for secrets managers. They are supposed to add an extra layer of security, but to me it seemed like they added much complexity but little benefit.
My though process went like this:
You either store secrets plain text on the server that your app can access or you store them encrypted somewhere else and fetch them at runtime.
But in order to fetch them you need to keep a secret on the server to fetch your secrets. So what's the whole point of this?
Secrets managers do not offer much additional security if someone ever gained access to your system. Storing the secret to fetch the secrets is pretty much the same as storing the secrets. But if someone has access to your system all hope is lost anyways.
The problem that secrets managers really solve
The point of secrets managers is not to secure your secrets on the server, but to secure them when they are not on the server!
There is nothing inherently wrong with using .env files. They can be used in a secure way, but it simply requires a lot more care and effort to do so.
If you think about it: if every collaborator has a copy of all secrets, how do you ensure that everyone treats them with care and no one makes a mistake? It's not an easy problem to fix.
At a certain scale dealing with .env files just becomes inconvenient to work with and it requires a lot of attention and maintenance to use them securely.
A secret manager can improve how secrets are shared within your team and remove the need to store them persistently on each collaborators machine in plain text.
On top secrets managers add some more convenient functionalities like versioning secrets, secrets rotation or granular access control to secrets.
The downsides of secrets managers
In a small project, this extra convenience might never be necessary. Or do you remember the last time, you needed granular access control for your API keys in your personal portfolio?
Using a secrets manager comes at a cost:
- increased complexity
- provider lock-in
- another party in the chain of trust
- performance overhead
- potential single point of failure
If you are the only one who ever touches your secrets, you rotate them by hand if necessary. And as long as you follow some basic security best practices, being mindful about your actions can get you quite far.
Summary
Security isn't a yes-no question.
Security is about reducing the possibilities something could go wrong at any stage of the process, from development to operations and beyond.
Adding a secrets manager provides extra guide rails that make it harder to mess up security. But it comes at a cost and there isn't an end to this...
You can spend unlimited time and resources on adding layers of bubble wrap, but there always will be a weak link in the chain. You always have to trust another human to act responsibly and there is no replacement for acting carefully.
For your personal portfolio I wouldn't bother introducing another dependency. But if you are dealing with a team of constantly rotating devs, the need for additional tooling to manage secrets will become apparent very quickly and at that point it will be obvious to you.
In the mean time, whether you add a secrets manager or not is really just a question of what you feel comfortable dealing with.
Top comments (9)
The best way to avoid vendor lock-in or dependency on the 3rd parties etc. is to custom build an asymmetric encryption library or a package that could be potentially utilized across the board within an organization.
Haha, good one! 😄
If you are afraid of lock-in or another party to trust, you can build your own. A secrets manager is basically a key-value store. If you only expose that to the networks you trust, it is like any other key-value store you add to a network.
I think a secrets manager shines when you are doing CI/CD. When deploys happen in multiple environments a secrets manager makes it easier to get the secrets for a specific project on a specific environment.
I agree it is a failure point, to prevent it from becoming one you could create an action/task in the CI/CD workflow that adds the secrets to the server.
Sometimes you need to see a tool for more than the advertised purpose. Just take the idea and see how it can benefit you.
You can build anything yourself, and sometimes a crappy version of software that already exists is enough, but it adds complexity and another headache to deal with... I think if developers tracked their time more precisely and had to pay their own hours, they probably think twice about entering these rabbit holes...
I see the CICD benefits though. To clarify, I am not entirely against using secrets managers (and in fact I use one) - but I think overengineering is real and there is a right time and place to do so...
With the rise of supply chain attacks it feels like security measures need to be more a focus than usual. They even changed devOps to devSecOps. Probably not for that reason, but I found it funny because that is an inherent part of the ops tasks.
You don't need to build something. It can be as easy as a private file on a cloud storage that is added to the server during a deploy. For added security the values can be encrypted.
If you have one project you can handle secrets manually. Once there are multiple projects with secrets I would look for the most barebones solution. And add features or look for a service when it is needed.
For me a secrets manager is more a convenience solution than a security solution.
You don't need to "keep a secret on your server to fetch your secrets" like you say. Are you unfamiliar with IAM permissions?
Sure, but it's pretty much the same thing... You don't store a secret yourself, but the machine comes preconfigured with a secret installed (that you don't see or set) that enables you to access secrets... IAM just means, Amazon (or whoever provider you are using) takes care of this step for you and does rotation etc. in the background.
I don't think that's correct... Admittedly I've never worked for Amazon/Google/Microsoft so I don't know exactly how their proprietary IAM/IMDS authentication works, and they don't exactly publish those details publicly. But I'm quite confident that they aren't storing static secrets on the server and then rotating them.
Somebody asked a question about how Azure does this over on StackOverflow and there's speculation in the answers that they are rotating certificates on the servers. But another answer points out that the cloud provider validates requests to the Identity Service through their own virtualized networking, which is not exactly something an attacker could spoof even if they had the hidden certificate. In other words, it's a little more complicated that just storing a secret on the server.
Yeah, I also don't exactly know how they do it, but you're right, probably not with a static secret... I was using the term "secret" more symbolically to say, that the server is authenticated to retrieve other secrets, be it a static secret, a certificate, through the virtual network, or whatever...
Also, not every provider offers IAM. I use Hetzner a lot, they are great! But their core offer is just plain VMs