Using Secrets in AWS Lambdas

In the previous part of the series, we had an overview of KMS, SSM Parameter Store, and Secrets Manager. In this part, let's put the services to the test and look at their different use cases with AWS Lambda.

As a big fan and long-time user of the serverless framework, we will also take a look at how storing secrets and the serverless framework play together.

Secrets in Environment Variables

In the Lambda the encrypted secrets may still be stored as environment variables, combining Secrets Manager, or Parameter Store with KMS.
That way environment variables still contain secrets but are stored in one central place. However, when you want to update a secret all functions using it must be redeployed. This is not practical.

Nonetheless, if you fancy this solution, make sure to avoid, that any user can see the unencrypted environment variables in the Lambda function's settings you deny them the kms:Decrypt permission.
You can deny users access to all KMS key decryption with the following permission:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "BlockKMSDecryptForAllResources",
            "Effect": "Deny",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
    ]
}

provider:
  kmsKeyArn: arn:aws:kms:us-east-1:XXXXXX:key/some-hash # KMS key arn which will be used for encryption for all functions

Serverless supports kmsKeyArn on provider level to be used by all functions or on per functions basis. It is used to encrypt your environment variables on AWS. Beware, that if you haven't encrypted your secrets with KMS, using for instance the AWS CLI, before deploying your function, the secrets get transported in plaintext.

In case you have encrypted them and only save their encrypted value in environment variables, you need to decrypt them inside of your Lambda function:

const AWS = require('aws-sdk');
AWS.config.update({ region: 'eu-west-1' });

const functionName = process.env.AWS_LAMBDA_FUNCTION_NAME;
const encrypted = process.env['MY_SECRET_ENV'];
let decrypted;

function processEvent(event) {
    // TODO handle the event here
}

exports.handler = async (event) => {
    if (!decrypted) {
        // Decrypt code should run once and variables stored outside of the
        // function handler so that these are decrypted once per container
        const kms = new AWS.KMS();
        try {
            const req = {
                CiphertextBlob: Buffer.from(encrypted, 'base64'),
                EncryptionContext: { LambdaFunctionName: functionName },
            };
            const data = await kms.decrypt(req).promise();
            decrypted = data.Plaintext.toString('ascii');
        } catch (err) {
            console.log('Decrypt error:', err);
            throw err;
        }
    }
    processEvent(event);
};

While it's certainly possible to protect your secrets by making them encrypted environment variables, it adds to the function's execution time. Also, there's no central place for you to manage all your secrets with this approach.

Secrets in Parameter Store or Secrets Manager

Still with Environment Variables

When you store your secrets within SSM Parameter Store or Secrets Manager, they are managed in one central place. This can make rotating secrets for numerous Lambdas a lot easier. However, it still depends on your usage of these services.

The serverless framework has built-in support for SSM parameters and Secrets Manager. You can get a SecureString, or other parameters, from SSM with ${ssm:/path/to/secureparam} inside your serverless.yml. To get a secret from Secrets Manager the syntax looks similar with ${ssm:/aws/reference/secretsmanager/secret_ID_in_Secrets_Manager}. Be aware, that SSM SecureString automatically gets decrypted and parsed if they export stringified JSON content. You can turn this off by, passing the raw instruction into variables (${ssm(raw):/path/to/secureparam}). You can even get a raw SSM parameter from another region with ${ssm(eu-west-1, raw):/path/to/secureparam}. Unless you pass the raw instruction your secrets get decrypted and stored in plaintext within your Lambda's environment variables. The only advantage to using purely KMS is that you manage your secrets in one central place. Not every developer needs to have access to the secrets to use them when working on one of your Lambdas. The problem remains, that the secrets get stored in environment variables. Therefore, they only change when a new deployment happens. Unless you use the raw instruction your secrets get stored in plaintext, and if you use the raw instructions you need to decrypt them within your Lambda with kms.decrypt().

No more secrets in Environment Variables

Instead of storing the secret in the environment variable, we just store the key from Secrets Manager/Parameter Store inside of it. You need to change your Lambda's code to fetch the secret from SSM and/or Secrets Manager, through this, the Lambda function always gets the secret directly from the Secrets Manager/Parameter Store. If we now change a secret it will be available to all functions right away, without having to change anything on the function itself. We could also get rid of the environment variables altogether and specify the secret keys inside our code.

The downside to the increased security and maintainability with this approach is that the latency to retrieve the secrets and decrypt them adds to the Lambda function's execution time. Therefore, it's recommended to combine this approach with a caching of the secrets, to avoid retrieving, decrypting, processing them every time from the start.

When you add a secret to Secrets Manager you can pick in the last step from a number of programming languages to get a code snippet for use in your function's code.

For a secret with name test/abc the code snippet would look like this:

// Use this code snippet in your app.
// If you need more information about configurations or implementing the sample code, visit the AWS docs:
// https://aws.amazon.com/developers/getting-started/nodejs/

// Load the AWS SDK
var AWS = require('aws-sdk'),
    region = "eu-west-1",
    secretName = "test/abc",
    secret,
    decodedBinarySecret;

// Create a Secrets Manager client
var client = new AWS.SecretsManager({
    region: region
});

// In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
// See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
// We rethrow the exception by default.

client.getSecretValue({SecretId: secretName}, function(err, data) {
    if (err) {
        if (err.code === 'DecryptionFailureException')
            // Secrets Manager can't decrypt the protected secret text using the provided KMS key.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InternalServiceErrorException')
            // An error occurred on the server side.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InvalidParameterException')
            // You provided an invalid value for a parameter.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InvalidRequestException')
            // You provided a parameter value that is not valid for the current state of the resource.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'ResourceNotFoundException')
            // We can't find the resource that you asked for.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
    }
    else {
        // Decrypts secret using the associated KMS CMK.
        // Depending on whether the secret is a string or binary, one of these fields will be populated.
        if ('SecretString' in data) {
            secret = data.SecretString;
        } else {
            let buff = new Buffer(data.SecretBinary, 'base64');
            decodedBinarySecret = buff.toString('ascii');
        }
    }

    // Your code goes here. 
});

While SSM does not provide you with a code snippet for parameters stored it will be somewhat similar to how you get secrets from Secrets Manager.

Instead of reinventing the wheel, I suggest having a look into Middy and see if the plugins for SSM Parameter Store and Secrets Manager fulfill your needs. The plugins come with built-in caching support. If you're working with TypeScript or JavaScript it's worth looking into Middy as it provides further useful plugins, and is easy to extend and adapt.

Conclusion

Don't save secrets in environment variables if you use the secret in other Lambdas as well. Instead, store them centrally in one place using SSM Parameter Store or Secrets Manager. Unless you need auto-rotation and multi-region replica support for your secrets SSM Parameter Store will likely be the perfect fit for you. Also, SSM provides a history of your parameter values.
Access the secrets in your Lambda using the fitting Middy plugin and cache the value for consecutive executions. How long you cache depends on your use case.

References

• https://dev.to/monisnap/hide-your-f-king-api-keys-and-credentials-from-versioned-code-183p
• https://medium.com/@furqanshaikh/securing-secrets-in-aws-lambda-93c12861371f
• https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html#configuration-envvars-encryption
• https://www.serverless.com/framework/docs/providers/aws/guide/serverless.yml/
• https://www.serverless.com/framework/docs/providers/aws/guide/variables/#reference-variables-using-the-ssm-parameter-store
• https://serverless-stack.com/chapters/storing-secrets-in-serverless-apps.html