The problem
How can code running in the managed AWS Lambda environment call services that use private certificates for HTTPS?
The majority of enterprises moving to AWS or other cloud platforms have existing on-premises applications, and there is often a need for the new cloud based applications to talk back to services on-prem. Typically this done with a hybrid network where the corporate network and AWS VPC(s) are connected using a VPN or Direct Connect.
Let's assume that the network has been set-up and the on-prem service is either using public DNS or a solution like Route 53 resolver has been configured and it's possible to establish a connection to the on-premises service.
If the service is using a public certificate (issued by a public certificate authority that the Lambda environment is aware of and has a copy of the root certificate public key) then there will be no problems, but if the enterprise is using their own private Certificate Authority then there will be an error making HTTPS calls.
Enterprises may have their own CA for services that were only ever expected to be used internally. It gives greater control over issuing and revoking certificates. Typically the CA certificate will be installed onto desktops and services at the enterprise to ensure that connections to these services are trusted in browsers and between services.
If this were an EC2 instance calling back to on-prem then you would probably install the certificate as part of the instance set-up, but we have less control in the transient Lambda environment.
It's possible to write code to handle the TLS failure, or even to disable certificate validation entirely (please don't do this), but what if there was a better way to have the lambda function transparently trust the private certificate?
The solution
Lambda Layers to the rescue!
Many TLS/SSL libraries or application frameworks have mechanisms to add additional root certificates to the "trust store". Once this is configured, any connections using certificates that were signed by the private root certificate will be automatically trusted.
On Linux (and macOS) .NET Core uses OpenSSL for cryptography and OpenSSL allows you to add additional root certificates from a file (in PEM format) using the SSL_CERT_FILE environment variable. The root certificate doesn't need to be "installed" into the environment.
Setting environment variables is easy for Lambda functions, but where should this certificate file be, and how do we get it into the Lambda environment?
The contents of the Lambda zip file are extracted to /var/task, and it's possible to include the certificate file here and point SSL_CERT_FILE to a location in this directory, however this has the drawback that every lambda zip file needs to contain the certificate file and you either need to include it into every repository, or include it as part of your CI/CD process.
Lambda Layers have two main use-cases: sharing dependencies (typically code or libraries, but can be configuration like this) or creating a custom runtime. The contents of the layer are extracted to /opt.
If we build a layer containing our certificate file it can be re-used across as many lambda functions as we like.
Building and sharing the layer
This SAM template will build a Certificate Lambda Layer including files in the certs sub-directory in the layer zip file.
The ARN for the layer is placed in a Parameter Store value that can be referenced by the templates for Lambda functions.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: SAM Template for certificate layer
Resources:
CertificateLayer:
Type: AWS::Serverless::LayerVersion
Properties:
CompatibleRuntimes:
- dotnetcore2.1
- dotnetcore3.1
ContentUri: ./certs/
Description: Layer containing additional certificates
RetentionPolicy: Retain
LayerParameter:
Type: AWS::SSM::Parameter
Properties:
Description: ARN for latest Certificate Layer
Name: /Lambda/Layers/Certificate
Type: String
Value: !Ref CertificateLayer
Within the certs sub-directory there should be a file containing one or more root certificates in PEM format. Typically this file will have a .crt extension, so let's call it additional-certificates.crt.
If we need to add additional root certificates (or remove expired ones), the layer can be updated, which will create a new version. Existing functions will continue to use the old version, but the Parameter Store key will now point to the ARN of the new version so any new deployments will pick up the change.
Why not a CloudFormation export? CloudFormation exports are good for things that never change, but you can't update the value of an export that's in use by another stack.
In the parameters of your CloudFormation stacks containing your Lambda functions, include a parameter to pull in the value from Parameter store and then use it with the function, also setting the SSL_CERT_FILE environment variable:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Lambda Function
Parameters:
CertificateLayer:
Default: /Lambda/Layers/Certificate
Description: Certificate Layer ARN
Type: AWS::SSM::Parameter::Value<String>
Resources:
MyFunction:
Type: AWS::Serverless::Function
Properties:
Environment:
Variables:
SSL_CERT_FILE: /opt/additional-certificates.crt
Layers:
- !Ref CertificateLayer
# Plus all the other properties
Once again we're using SAM here, although there is unfortunately an issue with the SAM CLI build command where it does not like the use of parameter store values for layers. There are some work arounds described in the GitHub issue.
Other runtimes
I've used this approach with .NET Core lambda functions, but this layer may be useful for other lambda runtimes.
For example, the popular Python requests library also allows you to configure certificates using the REQUESTS_CA_BUNDLE environment variable.
Top comments (7)
Thanks for the guide.
I'm running into a problem when I try to deploy my SAM app that suppose to use the CertificateLayer.
"Failed to create changeset for the stack: SamFromS3ToEH, An error occurred (ValidationError) when calling the CreateChangeSet operation: Unable to fetch parameters [XXX] from parameter store for this account."
XXX = I tried to change the parameter to "Certificate"/"CertificateLayer"/the arn of the certificate layer.
And I wonder how to fix it.
If you're having issues with this I would first try bypassing Parameter store and verify that if you use the ARN as a parameter (of type String) that it works correctly.
If that works then I wonder whether there is a permissions problem - does the user/role trying to create the changeset have full access to parameter store or are they limited to particular parameters. The ARNs for Parameter Store keys are a bit confusing because if you have a leading
/in the parameter store key it shouldn't be part of the ARN (i.e. it'sarn:aws:ssm:us-east-2:123456789012:parameter/Lambda/Layers/Certificaterather thanarn:aws:ssm:us-east-2:123456789012:parameter//Lambda/Layers/Certificate.If it's a permissions problem you might be able to find more information in CloudTrails about the specific permissions problem - and it might not be the parameter itself - if you're using a SecretString rather than String type it could be KMS permissions.
Thank you very much for sharing. I made the mistake of not using the exact SSL_CERT_FILE environment variable and named my variable as sslCertFile = /opt/certificate.crt. The SSL_CERT_FILE environment variable must be used in its exact context.
Thank you for the comment. But I need to ask you if I want to add multiple certificates what should I do.
I tried to add them like this: SSL_CERT_FILE:/opt/sknkqw.cer
SSL_CERT_FILE2:/opt/hfbeh.cer
This sample do not provide a view how to use a certificate inside a code. After translating into serverless yml I still can not say is this article useful for me.
The reason for configuring this layer is so that you do not need to use the certificate programmatically within your code - your .NET Core lambda code will automatically be able to make HTTPS calls to services using private certificates (e.g. using HttpClient or other libraries) without you needing to add any special handling for private certificates into your codebase. Think of this as being equivalent to installing the certificate on a Windows server to prevent these types of TLS errors occurring.
Thank you very much for sharing this article. It solved my problem.