DEV Community: Stephan

Using Secrets in AWS Lambdas

Stephan — Wed, 30 Jun 2021 08:41:15 +0000

In the previous part of the series, we had an overview of KMS, SSM Parameter Store, and Secrets Manager. In this part, let's put the services to the test and look at their different use cases with AWS Lambda.

As a big fan and long-time user of the serverless framework, we will also take a look at how storing secrets and the serverless framework play together.

Secrets in Environment Variables

In the Lambda the encrypted secrets may still be stored as environment variables, combining Secrets Manager, or Parameter Store with KMS.
That way environment variables still contain secrets but are stored in one central place. However, when you want to update a secret all functions using it must be redeployed. This is not practical.

Nonetheless, if you fancy this solution, make sure to avoid, that any user can see the unencrypted environment variables in the Lambda function's settings you deny them the kms:Decrypt permission.
You can deny users access to all KMS key decryption with the following permission:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "BlockKMSDecryptForAllResources",
            "Effect": "Deny",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
    ]
}

provider:
  kmsKeyArn: arn:aws:kms:us-east-1:XXXXXX:key/some-hash # KMS key arn which will be used for encryption for all functions

Serverless supports kmsKeyArn on provider level to be used by all functions or on per functions basis. It is used to encrypt your environment variables on AWS. Beware, that if you haven't encrypted your secrets with KMS, using for instance the AWS CLI, before deploying your function, the secrets get transported in plaintext.

In case you have encrypted them and only save their encrypted value in environment variables, you need to decrypt them inside of your Lambda function:

const AWS = require('aws-sdk');
AWS.config.update({ region: 'eu-west-1' });

const functionName = process.env.AWS_LAMBDA_FUNCTION_NAME;
const encrypted = process.env['MY_SECRET_ENV'];
let decrypted;

function processEvent(event) {
    // TODO handle the event here
}

exports.handler = async (event) => {
    if (!decrypted) {
        // Decrypt code should run once and variables stored outside of the
        // function handler so that these are decrypted once per container
        const kms = new AWS.KMS();
        try {
            const req = {
                CiphertextBlob: Buffer.from(encrypted, 'base64'),
                EncryptionContext: { LambdaFunctionName: functionName },
            };
            const data = await kms.decrypt(req).promise();
            decrypted = data.Plaintext.toString('ascii');
        } catch (err) {
            console.log('Decrypt error:', err);
            throw err;
        }
    }
    processEvent(event);
};

While it's certainly possible to protect your secrets by making them encrypted environment variables, it adds to the function's execution time. Also, there's no central place for you to manage all your secrets with this approach.

Secrets in Parameter Store or Secrets Manager

Still with Environment Variables

When you store your secrets within SSM Parameter Store or Secrets Manager, they are managed in one central place. This can make rotating secrets for numerous Lambdas a lot easier. However, it still depends on your usage of these services.

The serverless framework has built-in support for SSM parameters and Secrets Manager. You can get a SecureString, or other parameters, from SSM with ${ssm:/path/to/secureparam} inside your serverless.yml. To get a secret from Secrets Manager the syntax looks similar with ${ssm:/aws/reference/secretsmanager/secret_ID_in_Secrets_Manager}. Be aware, that SSM SecureString automatically gets decrypted and parsed if they export stringified JSON content. You can turn this off by, passing the raw instruction into variables (${ssm(raw):/path/to/secureparam}). You can even get a raw SSM parameter from another region with ${ssm(eu-west-1, raw):/path/to/secureparam}. Unless you pass the raw instruction your secrets get decrypted and stored in plaintext within your Lambda's environment variables. The only advantage to using purely KMS is that you manage your secrets in one central place. Not every developer needs to have access to the secrets to use them when working on one of your Lambdas. The problem remains, that the secrets get stored in environment variables. Therefore, they only change when a new deployment happens. Unless you use the raw instruction your secrets get stored in plaintext, and if you use the raw instructions you need to decrypt them within your Lambda with kms.decrypt().

No more secrets in Environment Variables

Instead of storing the secret in the environment variable, we just store the key from Secrets Manager/Parameter Store inside of it. You need to change your Lambda's code to fetch the secret from SSM and/or Secrets Manager, through this, the Lambda function always gets the secret directly from the Secrets Manager/Parameter Store. If we now change a secret it will be available to all functions right away, without having to change anything on the function itself. We could also get rid of the environment variables altogether and specify the secret keys inside our code.

The downside to the increased security and maintainability with this approach is that the latency to retrieve the secrets and decrypt them adds to the Lambda function's execution time. Therefore, it's recommended to combine this approach with a caching of the secrets, to avoid retrieving, decrypting, processing them every time from the start.

When you add a secret to Secrets Manager you can pick in the last step from a number of programming languages to get a code snippet for use in your function's code.

For a secret with name test/abc the code snippet would look like this:

// Use this code snippet in your app.
// If you need more information about configurations or implementing the sample code, visit the AWS docs:
// https://aws.amazon.com/developers/getting-started/nodejs/

// Load the AWS SDK
var AWS = require('aws-sdk'),
    region = "eu-west-1",
    secretName = "test/abc",
    secret,
    decodedBinarySecret;

// Create a Secrets Manager client
var client = new AWS.SecretsManager({
    region: region
});

// In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
// See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
// We rethrow the exception by default.

client.getSecretValue({SecretId: secretName}, function(err, data) {
    if (err) {
        if (err.code === 'DecryptionFailureException')
            // Secrets Manager can't decrypt the protected secret text using the provided KMS key.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InternalServiceErrorException')
            // An error occurred on the server side.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InvalidParameterException')
            // You provided an invalid value for a parameter.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'InvalidRequestException')
            // You provided a parameter value that is not valid for the current state of the resource.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
        else if (err.code === 'ResourceNotFoundException')
            // We can't find the resource that you asked for.
            // Deal with the exception here, and/or rethrow at your discretion.
            throw err;
    }
    else {
        // Decrypts secret using the associated KMS CMK.
        // Depending on whether the secret is a string or binary, one of these fields will be populated.
        if ('SecretString' in data) {
            secret = data.SecretString;
        } else {
            let buff = new Buffer(data.SecretBinary, 'base64');
            decodedBinarySecret = buff.toString('ascii');
        }
    }

    // Your code goes here. 
});

While SSM does not provide you with a code snippet for parameters stored it will be somewhat similar to how you get secrets from Secrets Manager.

Instead of reinventing the wheel, I suggest having a look into Middy and see if the plugins for SSM Parameter Store and Secrets Manager fulfill your needs. The plugins come with built-in caching support. If you're working with TypeScript or JavaScript it's worth looking into Middy as it provides further useful plugins, and is easy to extend and adapt.

Conclusion

Don't save secrets in environment variables if you use the secret in other Lambdas as well. Instead, store them centrally in one place using SSM Parameter Store or Secrets Manager. Unless you need auto-rotation and multi-region replica support for your secrets SSM Parameter Store will likely be the perfect fit for you. Also, SSM provides a history of your parameter values.
Access the secrets in your Lambda using the fitting Middy plugin and cache the value for consecutive executions. How long you cache depends on your use case.

References

Storing Secrets in AWS

Stephan — Wed, 30 Jun 2021 08:41:05 +0000

Secrets can be keys, authorization tokens, passwords, and everything else that should be kept private and in this context be represented as a string.

We can store secrets in multiple ways:

Encrypting environment variables
Store them in an encrypted file on S3
Storing secrets in dedicated service

We will look into each of these options shortly but before let's be more specific on "Storing secrets in dedicated service". There are numerous products available to store secrets, i.e. HashiCorp's Vault, or strongDM to name a few, however, here we just want to focus on what solutions AWS offers. Therefore, our "dedicated services" include the Secrets Manager and the System Manager Parameter Store.

Encrypting environment variables

AWS Key Management Service (AWS KMS) is an encryption and key management service scaled for the cloud. AWS KMS keys and functionality are used by other AWS services, and you can use them to protect data in your own applications that use AWS.

KMS is the ideal tool to create and manage cryptographic keys which can be used to encrypt and decrypt secrets. KMS is so deeply rooted within AWS that it's used when encrypting files on S3 or anywhere AWS offers encrypted storage. Important to note is that KMS only stores the key used for decrypting/encrypting the secret, not the secret itself. In the context of AWS Lambda, the secret would still be stored within an environment variable and be decrypted during the Lambda runtime adding to your execution time and bill. As KMS is just meant for encrypting and decrypting the secrets, it has no built-in mechanism to rotate secrets. With the secrets themselves, still stored as environment variables, rotating the secrets, you have to encrypt the secret before you store it in an environment variable. You have to repeat this step for every function using the secret and for every encryption key used, in case you use more than one. Also, keep in mind that KMS encrypts the environment variables at rest, meaning when you deploy a Lambda function, and the secrets are "in transit" they will not be encrypted.

Store them in an encrypted file on S3

While this may suffice at first and is very easy to implement, it is hard to keep your secrets secret and not be over-permissive or have other resources, i.e. Lambda functions, access secrets they are not using. I wouldn't recommend it.

Storing secrets in AWS System Manager Parameter Store

Parameter Store, a capability of AWS Systems Manager, provides secure, hierarchical storage for configuration data management and secrets management. You can store data such as passwords, database strings, Amazon Machine Image (AMI) IDs, and license codes as parameter values. You can store values as plain text or encrypted data. You can reference Systems Manager parameters in your scripts, commands, SSM documents, and configuration and automation workflows by using the unique name that you specified when you created the parameter.

Storing secrets inside Parameter Store as key-value pairs leverage KMS to encrypt the secrets at rest. A parameter is a key-value pair, where the value can be one string, a list of strings, or even a JSON object in string representation. Parameter Store allows to store SecureStrings encrypting them with KMS, but its default is to store plaintext parameter values.

The parameter key should follow a unique structure accepted by all admins of the Parameter Store within your account. The key must start with a slash (/) and could be followed by your department or team name, the service to which the secret belongs, and a parameter name e.g. /acme/serviceA/username. One reason for this is to limit access to parameters by key paths (/acme/serviceA/*). Another one is, that Parameter Store allows us to read all parameters within a hierarchy. With this, we can get all parameters to serviceA by retrieving all parameter keys starting with /acme/serviceA/.

Parameter Store does not charge anything for storing parameters with up to 4 kB size, if you want to store larger parameters you get charged 0.05 USD per parameter and month.

Rolling back secrets in Parameter Store is a piece of cake because Parameter Store provides a history of the parameter where you can not only see the previous value of the secret but also who changed it.

Together with CloudTrail this provides you with full control over your parameter stores. As Parameter Store is not region agnostic, the parameters get stored in a specific region. You can use them from other regions, of course, but if you want to replicate your parameters across regions, there's no built-in way to do so. However, with CloudTrail you could have a Lambda executed every time a parameter got changed.

Storing secrets in AWS Secrets Manager

AWS Secrets Manager helps you to securely encrypt, store, and retrieve credentials for your databases and other services. Instead of hardcoding credentials in your apps, you can make calls to Secrets Manager to retrieve your credentials whenever needed. Secrets Manager helps you protect access to your IT resources and data by enabling you to rotate and manage access to your secrets.

AWS Secrets Manager complies with many international security standards. The service is dedicated to storing all kinds of secrets and supports a default set of supported resources for which secret management is even simpler. Therefore, everything within Secrets Manager is stored encrypted with a KMS key.

You can store secrets as key-value pairs. A secret can consist of multiple key-value pairs. This is especially useful as you get charged 0.40 USD per stored secret and month. When you want to store an API key, username, password, and maybe even the URL, storing them in separate secrets would cost you 1.60 USD, whereas, if you store them together, similar to a JSON object you just pay 0.40 USD. Take care, that you need to parse the object, however.

For the secret's name, it's also recommended to follow some structure, though retrieving secrets hierarchically is not possible like it is in Parameter Store, you can still benefit from a somewhat structured key when setting up IAM permissions.
Like in other AWS services a tag can be helpful in the context of billing, i.e. separating the bill by a specific tag.

Resource permissions are particularly helpful when you want to share secrets across AWS accounts.

Similar to Parameter Store, Secrets Manager keeps the secrets within the AWS region they were created in. With "Replicate Secrets" you have the option to automatically have your secrets copied to other regions as read-only copies. The secret in the origin region is the only one you can edit. Read the getting started guide to learn more.

For your own secret types (API key, username, passwords, ...) Secret Manager does not provide automatic secret rotation. To rotate your secrets in this case, set up a Lambda function dedicated to creating new secrets and invalidating the old ones. Be careful in this Lambda function on what you log, to not accidentally leak your secrets. You can renew your secrets once a day or up to once a year, it's up to your use case.

When setting up Secrets Manager please consider the best practices.

Conclusion

You can mix and match KMS with Secrets Manager or Parameter Store or only use one of the services to keeping the secrets completely safe.
Secrets Manager and Parameter Store are in many parts very similar to one another. Parameter Store will likely be your cheaper solution. Unless you need an automatic rotation of secrets or secret replication across regions and fancy convenience, Parameter Store will likely be your preferred solution. It does not have to be one or the other as Secrets Manager and Parameter Store can be used side by side. Make sure to take advantage of CloudTrail and set up carefully drafted IAM roles for your users.

In the next part of the series, we look into how we can use these services to secure secrets when using AWS Lambda.

References

The case of monitoring your internet speed during a pandemic

Stephan — Sun, 24 Jan 2021 16:42:48 +0000

TL;DR: Not interested in the background and just want to know how to monitor your internet speed regularly? Jump right to it

I live in Germany, and I know that we don't have the fastest internet in the world. Still, with a mean download speed of 42.33 Mbps, the country does better than the global average of 24.83 Mbps. Nevertheless, it is just enough to place 42^nd out of 221 countries compared.¹ In comparison with only the European countries, Germany does not even make it into the top 10, placing 26^th.¹

With digitization as one of the thriving factors for the world economy, the internet and connectivity to it start playing an ever more important role, as it is the foundation that digitization needs to flourish.²

Currently, Germany is the 13^th biggest economy in the world - placing 8^th within Europe - measured by the gross domestic product (GDP) per capita, adjusted with purchasing power parity.³ So, one could expect to have a fast, reliable internet connection. At least, the comparably high economic strength of Germany ($ 56K) does not seem to have a significant effect on the price of an internet contract. However, it is noteworthy that Luxembourg, as the country with the strongest economic power within Europe ($ 121K) gets better value (internet speed of avg. 118 Mbps) for a lower price.⁴ Given that Luxembourg is a lot smaller than Germany, the countries face different challenges when it comes to getting the internet to where the people live. Therefore, it is an interesting fact, but one to be neglected.

Lately, the pandemic urges people to work from home wherever possible, and with schools and universities closed, the internet infrastructure in Germany is put to a test. During the day employees, students, pupils working from home put an unprecedented load on the infrastructure. From the afternoon to the evening and night, when people stop working and switch their internet usage to scrolling through Instagram, TikTok, etc., and streaming their favorite shows on Netflix, Prime, and so on.

The additional stress this puts on the internet service providers (ISPs) can be felt by customers. In my case, living in a densely populated area (a.k.a city), the result was that I had no internet connection at home between 8 a.m. and 11 p.m., making it impossible for me to work from home, or stream anything. I have a contract promising up to 50 Mbps and no less than 25 Mbps. For the past years, this was sufficient and I never had any serious problems with it. Now, in the middle of the second "lockdown", I effectively have no internet during the day.

Getting curious to see how much my internet speed varies throughout the day (and night) gave me the idea for this post.

Monitoring internet speed

While there are numerous tools out there to test your internet speed⁵, Ookla's speedtest CLI seemed like the perfect fit. You can simply download it, it's well documented, it requires no additional tools, it comes from the official source, and seems to be maintained in contrast to others⁶. Running the speed test from my PC was a no go. First of all, the test would run on my WiFi, possibly distorting the results. Secondly, it would not monitor the connection day and night, but only run when my PC is on, so only when I work. Furthermore, it would likely distort the results more, as I depend on the internet, namely Stackoverflow, to work. Luckily, I have a RaspberryPi connected to my router. Services already deployed on the RaspberryPi, run only as Docker containers. Therefore, I aimed to run the speed test from a Docker container too and to keep the OS free from all that is unnecessary.

The final result can be found on GitHub:

sthuber90 / docker-speedtest

Regularly test your internet speed with Speedtest CLI within Docker container

The repository contains the code, that is used to build the sthuber90/docker-speedtest Docker image. The container can be configured to run a speed test every 5 minutes (default: 15 minutes). Be aware, that if you run this image, you agree to the Speedtest CLI's license and privacy regulations.

The results of the speed test are intended to be forwarded to an InfluxDB and visualized in a Chronograf dashboard (provided in the repository). Detailed instructions to set up the monitoring can be found in the README on GitHub and Docker Hub.

I hope you did find this post helpful.

https://www.cable.co.uk/broadband/speed/worldwide-speed-league/, 2020 ↩
PwC, https://www.strategyand.pwc.com/m1/en/reports/digitization-for-economic-growth-and-job-creation.pdf, 2013 ↩
GDP from 2019, considering PPP (World Bank: https://data.worldbank.org/indicator/NY.GDP.MKTP.PP.CD?end=2019&most_recent_value_desc=true&start=1990&view=chart) turned into GDP per capita in USD, taking the countries population into account (World Bank: https://data.worldbank.org/indicator/SP.POP.TOTL?most_recent_value_desc=true), 2019 ↩
https://www.cable.co.uk/broadband/pricing/worldwide-comparison/, 2020 ↩
https://fast.com (by Netflix), http://avm.de/nc/service/zack-der-speedtest-fuer-ihre-breitbandverbindung/, https://breitbandmessung.de/ (by the German Federal Network Agency) ↩
Other libraries using https://speedtest.net for internet speed tests, that seem not to be maintained anymore (1) https://www.npmjs.com/package/speedtest, (2) https://pypi.org/project/speedtest-cli/ ↩