"CloudFront deployments with Lambda@Edge"

A/B Testing, Blue/Green deployments, Canary releases. Different, but still so much in common. They all have different purpose but are using basically using the same technical solution under the hood. We do these kind of tests for several reasons. One is to do a A/B test and determine what version our users like the best. Sure it is possible to do A/B testing on the client side, but personally I find it easier to do server side.

Blue / Green and Canary deployments are done to make sure a new version of the application work as we expect and give us an easy way to roll back to previous version in case of a problem. All of these are important practices in the DevOps culture.

Many services from AWS offer this solution out of the box, one service that doesn't is CloudFront. Luckily CloudFront has the possibility to run Lambda@Edge which we can use to solve this.

All source code for this setup is found on GitHub

Introducing Lambda@Edge

Lambda can run in four different locations in the request flow.

Viewer request - Is run when CloudFront receives a request from a viewer.

Origin request - Is run before CloudFront forwards a request to the origin.

Origin response - Is run when CloudFront receives a response from the origin.

Viewer response - Is run before CloudFront returns the response to the viewer.

There are several limitations when it comes to Lambda@Edge, it is only possible to create functions in Python and Node.js. Viewer request and response functions can only allocate 128mb of memory and only run for 3 seconds. You can't use environment variables, you can't use the latest version alias, only fixed versions are supported. Logs are published to the edge region that you access and not to us-east-1 region, even if functions must be deployed to that region. Be sure to read the documentation before you start working with Lambda@edge.

Solution Overview

In this solution we are going to use two different Lambda functions for viewer request and origin response hooks, we will store configuration in Parameter Store, and S3 is our origin.

In the viewer request hook we will check for a special cookie, if the cookie is not set we will fetch a random version. In the origin response we set the set-cookie header to store the version we are using.

By setting different values in Parameter Store we can control the weight of each version and we can also reset and have clients receive new random version.

Viewer Request

Let's start with the Lambda function that will react to the Viewer Request Event. This Lambda function is responsible for checking for our cookie X-Version-Name if an value is set the function will update the request path based on the cookie value. If there is no value set it will roll the dice and get a random version and update the request path. This function will also match the value in cookie X-Version-Reset towards a value in Parameter Store and if the value is different it will ignore any set value in X-Version-Name. If there is no value in X-Version-Name or if the value is ignored the function throws the dice and picks a version at random. The weight, for the different versions, are controlled by a value in Parameter Store. Finally the function pass the cookie to the next step in the call chain.

The Viewer Request code

Full version is available in GitHub.

def lambda_handler(event, context):
    request = event['Records'][0]['cf']['request']
    headers = request['headers']

    cookie_version_blue = 'X-Version-Name=Blue'
    cookie_version_green = 'X-Version-Name=Green'
    path_blue = '/Blue'
    path_green = '/Green'

    uri = ''

    if request['uri'].endswith('/'):
        request['uri'] = request['uri'] + 'index.html'

    if 'cookie' not in request['headers']:
        request['headers']['cookie'] = []

    # Reset weights, ignore already set cookie
    reset_weight, reset_cookie = do_weight_reset(headers)
    if not reset_weight:
        for cookie in headers.get('cookie', []):
            if cookie_version_blue in cookie['value']:
                uri = path_blue + request['uri']
                break
            elif cookie_version_green in cookie['value']:
                uri = path_green + request['uri']
                break
    request['headers']['cookie'].append(
        {'key': 'Cookie', 'value': reset_cookie})

    if not uri:
        weight = int(load_parameter('Weight'))
        cookie_value = ''

        if random.random() < float(weight / 100.0):
            uri = path_blue + request['uri']
            cookie_value = cookie_version_blue
        else:
            uri = path_green + request['uri']
            cookie_value = cookie_version_green
        request['headers']['cookie'].append(
            {'key': 'Cookie', 'value': cookie_value})

    request['uri'] = uri
    return request

Viewer Response

The Viewer Response function basically has one task, and that is to pass set-cookie header to the client. This is needed so the client set the cookies X-Version-Name and X-Version-Reset so the client send them in the next request. This is important so the client doesn't jump between versions. The function has a small but very important job to do.

The Viewer Response code

Full version is available in GitHub.

def lambda_handler(event, context):
    response = event['Records'][0]['cf']['response']
    request = event['Records'][0]['cf']['request']
    # Persist cookie, set the set-cookie header
    if 'set-cookie' not in response['headers']:
        response['headers']['set-cookie'] = []

    request_headers = request['headers']
    cookie_version_blue = 'X-Version-Name=Blue'
    cookie_version_green = 'X-Version-Name=Green'
    cookie_reset = 'X-Version-Reset'

    for cookie in request_headers.get('cookie', []):
        if cookie_version_blue in cookie['value']:
            response['headers']['set-cookie'].append(
                {'key': 'set-cookie', 'value': cookie_version_blue})
        elif cookie_version_green in cookie['value']:
            response['headers']['set-cookie'].append(
                {'key': 'set-cookie', 'value': cookie_version_green})
        elif cookie_reset in cookie['value']:
            response['headers']['set-cookie'].append(
                {'key': 'set-cookie', 'value': cookie['value']})

    return response

Deploying the functions

The Lambda functions need to be deployed in us-east-1 region, since that is the region Lambda@Edge originates from. We must also use a fixed version and can't use the latest alias. As normal AWS SAM is used to define and deploy Lambda Functions.

The SAM Template

Full version is available in GitHub.

ViewerRequestFunction:
    Type: AWS::Serverless::Function
    Properties:
        AutoPublishAlias: "true"
        Runtime: python3.7
        MemorySize: 128
        Timeout: 3
        CodeUri: ./viewer-request
        Handler: handler.lambda_handler
        AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
            - Effect: Allow
            Principal:
                Service:
                - lambda.amazonaws.com
                - edgelambda.amazonaws.com
            Action:
                - sts:AssumeRole
        Policies:
        - SSMParameterReadPolicy:
            ParameterName: !Sub ${SsmConfigPath}/*
        - Version: "2012-10-17"
            Statement:
            Action:
                - lambda:GetFunction
            Effect: Allow
            Resource: "*"

ViewerResponseFunction:
    Type: AWS::Serverless::Function
    Properties:
        AutoPublishAlias: "true"
        Runtime: python3.7
        MemorySize: 128
        Timeout: 3
        CodeUri: ./viewer-response
        Handler: handler.lambda_handler
        AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
            - Effect: Allow
            Principal:
                Service:
                - lambda.amazonaws.com
                - edgelambda.amazonaws.com
            Action:
                - sts:AssumeRole
        Policies:
        - Version: "2012-10-17"
            Statement:
            Action:
                - lambda:GetFunction
            Effect: Allow
            Resource: "*"

CloudFront setup

Finally we need to create the CloudFront distribution and set the Lambda functions for the Viewer Request and Response triggers. Normally CloudFront will not use and pass the headers to the cache. Since our setup is depending on two cookies we must make sure CloudFront pass them along. That is done by adding them to WhitelistedNames section. Wildcards are supported so we just add X-Version-* to that section.

The CloudFormation Template

Full version is available in GitHub.

CloudFrontDistribution:
    Type: AWS::CloudFront::Distribution
    Properties:
      DistributionConfig:
        Comment: !Sub "Distribution for ${ProjectName}"
        DefaultCacheBehavior:
          AllowedMethods:
            - "GET"
            - "HEAD"
            - "OPTIONS"
          Compress: False
          DefaultTTL: 0
          MaxTTL: 0
          MinTTL: 0
          ForwardedValues:
            QueryString: False
            Cookies:
              Forward: whitelist
              WhitelistedNames:
                - "X-Version-*"
          LambdaFunctionAssociations:
            - !If
              - ViewerRequestLambdaArnSet
              - EventType: viewer-request
                LambdaFunctionARN: !Ref ViewerRequestLambdaArn
              - !Ref AWS::NoValue
            - !If
              - ViewerResponseLambdaArnSet
              - EventType: viewer-response
                LambdaFunctionARN: !Ref ViewerResponseLambdaArn
              - !Ref AWS::NoValue
          TargetOriginId: !Sub ${ProjectName}-origin
          ViewerProtocolPolicy: redirect-to-https
        DefaultRootObject: !Ref DefaultRootObject
        Enabled: True
        Origins:
          - DomainName: !Sub ${StorageBucket}.s3.amazonaws.com
            Id: !Sub ${ProjectName}-origin
            S3OriginConfig:
              OriginAccessIdentity: !Sub origin-access-identity/cloudfront/${OriginAccessIdentity}
        PriceClass: PriceClass_100
      Tags:
        - Key: Name
          Value: !Sub ${ProjectName}

Conclusion

Even though CloudFront doesn't support these deployments techniques out of the box Lambda, as so many times before, come to the rescue. The versatility of AWS Lambda is truly a miracle! Download the code and take it for spin!

Happy hacking!