Secrets in Lambda Env Vars

Hello, friends!

I'm here on this fine Thursday with some low-key advice on a very serious problem: you—or someone you know—has been putting secrets into Lambda environment variables. We've all done it, and there's no shame in learning, but it does need to stop. I'm here to help you think through some alternatives.

The problem

If you're a user of AWS Lambda, and you need to give your Lambda an API key to your Quietscheentchen account, there's a good chance that you've done what many in the past have done with that API key:

1. Put the secret in SSM
2. Reference it from Serverless or CloudFormation
3. Deployed it as an environment variable on the Lambda

There's a big problem with this, though, and it's not always obvious at first. Say you're incorporating a new service which monitors the configuration of your Lambdas in order to help you reduce the amount you spend on your Lambdas overall. That vendor needs you to set them up in your AWS account:

• Give them an IAM role, with external ID, so that they can access your account
• Because you're following the principle of least privilege, you only give them the ability to call the GetFunction API in AWS

Here's the problem, though: environment variables are returned by this API call in plaintext, even if you used a secure string SSM parameter when you were deploying the Lambda.

This is a problem for three major reasons:

1. Your new vendor now has the API token
2. If ever you need to rotate that API key (which you should immediately do once you realize you exposed it to your vendor), you will have to re-deploy your Lambdas in order to update the key
3. Every other person with AWS Console access which can see that Lambda now can also see that API key

Even if you trust your wonderful new vendor, their having the ability to see that key means that if they suffer a breach then their attacker may also now have your API key.

All vendors have breaches, and your new vendor is no different.

Paths forward

Lucky for you, you're clever and you've got options! Here are some of them.

Use AWS Secrets Manager instead

The first option is to move your secrets to AWS Secrets Manager. You can do this by changing your code to fetch the secrets from AWS Secrets Manager at runtime, and then using them from there. If your code has too many references to the environment variable, no problem! All you need to do is have your program fetch them and then place them in the environment for your code to access.

This will also allow you to rotate the secret at will without having to re-deploy, and then you can also use AWS IAM permissions to restrict access to the secret.

Bonus points: you can automate the token rotation here and never manually rotate a token again!

Use a 1Password Service Account

Using 1Password Service Accounts is a good option if you're operating in a multi-cloud ecosystem, or if you need to share the secrets in many different contexts.

The overall process is the same as for Secrets Manager, except that you use 1Password Service Accounts to access the secrets in that vault rather than using AWS Secrets Manager.

Use SSM at Runtime

Same as Secrets Manager, but use SSM Secure Strings, but fetch them at runtime rather than referencing them for environment variables.

Wrapping Up

As we move forward as professionals, we owe it to ourselves, our organizations, our customers, and our users to improve our security game. I hope that these tips are helpful, and I look forward to seeing your thoughts in the comments.

Thanks!