CloudFront Functions - dynamic switching between origins

You can switch between different applications sharing the same URL in CloudFront.

The use case

I have a web app served using CloudFront from S3. I need a mechanism to seamlessly replace this app with another one, served from a different S3 bucket, but with the same URL. The bottom line is to switch apps under the hood without redeploying the stack so users won't really notice a switch.

Architecture

Code is available in this repository

There are a few ways to skin the cat. I decided to use a CloudFront Function to dynamically decide which origin to use. The parameter is to be stored in the associated CloudFront KeyValueStore.

When working with requests and responses in CloudFront, there are two main candidates: CloudFront Functions and Lambda@Edge. The latter allows running Node.js or Python Lambda runtime close to the user's call. Lambda@Edge can connect to AWS services and, in general, it behaves as a regular Lambda, for better or worse.
CloudFront Function is restricted to the JavaScript runtime and doesn't have access to third-party services, but compensates for these limitations with low latency.

Besides that, probably the biggest difference is when different function types are called. Lambda@Edge is invoked only on cache misses, and the CloudFront Function is invoked on every request. In my case, it is not a problem, but for some traffic patterns it might be problematic to run the CloudFront Function every time.

Implementation

Initial setup

I start by creating a dummy setup with CloudFront and two web apps: AppOne and AppTwo. They have separate behaviors defined and separate S3 origins.

App One:

App Two:

The default behavior points to AppOne:

This is a view in the console:

CloudFront Function

I am going to update the default behavior so it points to App One or Two, depending on the defined parameter.

I create a CloudFront Function. It is expected to be very simple and can be written inline in the template. From what I understood, this is the only way to use dynamic parameters in the function body.

OriginSwitchFunction:
    Type: AWS::CloudFront::Function
    Properties:
      FunctionCode: !Sub
      - |
        import cf from 'cloudfront'
        const kvsId = '${kvsId}'
        const appOneOrigin = '${appOneOrigin}'
        const appTwoOrigin = '${appTwoOrigin}'
        const kvsHandle = cf.kvs(kvsId)
        async function handler(event) {
          let originId = appOneOrigin
          try {
            const value = await kvsHandle.get("app-version");
            console.log("app-version: " + value)
            if (value === 'appTwo') {
              originId = appTwoOrigin
            }
          } catch (e) {
            console.log("error when getting a key: " + e.message)
          }
          cf.selectRequestOriginById(originId)
          return event.request
        }
      - kvsId: !GetAtt OriginSwitchKvs.Id
        appOneOrigin: AppOneOrigin
        appTwoOrigin: AppTwoOrigin
      FunctionConfig:
        KeyValueStoreAssociations:
          - KeyValueStoreARN: !GetAtt OriginSwitchKvs.Arn
        Comment: Function for origin switching
        Runtime: cloudfront-js-2.0
      Name: origin-switch
      AutoPublish: true

To have it working, I also create a KeyValueStore and associate it with the function.

  OriginSwitchKvs:
    Type: AWS::CloudFront::KeyValueStore
    Properties:
      Name: origin-switch

And the last step is to associate the function with the given behavior. In this example, I am using the default behavior

        DefaultCacheBehavior:
          FunctionAssociations:
            - FunctionARN: !GetAtt OriginSwitchFunction.FunctionARN
              EventType: viewer-request
          ForwardedValues:
            QueryString: true
            Headers:
              - X-Original-Host
          AllowedMethods:
            - GET
            - HEAD
            - OPTIONS
          ViewerProtocolPolicy: allow-all
          TargetOriginId: AppOneOrigin

Test

Once the template is deployed, I can start testing it. The function looks good, and it has KeyValueStore association. I can test it manually in the console

I set the parameter to appOne and open the page:

Looks OK. Now let's update the parameter and open the page once more:

Hmm, it is not right.

The answer is in the bottom right corner:

CloudFront uses caching based on the path and doesn't care about the origin used. In other words, the CloudFront Function decides which origin to use, but then if there is a cache hit, the stored version is returned. And this version might belong to the old origin.

The solution is to run cache invalidation every time the parameter is updated. After invalidating the cache, the app version is returned as expected.

Updating script

I created a script that updates KeyValueStore and triggers cache invalidation.

The issue I started to see was that even after cache invalidation, sometimes the previous version of the app was returned to the browser. I've ruled out local browser caching. Probably the distributed nature of CloudFront and its KeyValueStore has been causing some race conditions.

As I mentioned before, the caching mechanism of CloudFront doesn't take into consideration which origin is used under the hood, as long as the path matches the cached object. This can be overcome by using custom headers. This way CloudFront creates a cache for each header version - link to docs

Here is a fixed version of the function:

# ...
cf.selectRequestOriginById(originId)
          event.request.headers['x-app-version'] = { value: originId }
          return event.request
# ...

and updated configuration for the default behavior:

        DefaultCacheBehavior:
          FunctionAssociations:
            - FunctionARN: !GetAtt OriginSwitchFunction.FunctionARN
              EventType: viewer-request
          ForwardedValues:
            QueryString: true
            Headers:
              - X-Original-Host
              - X-App-Version

Now I can run the script. Eventually, I would keep it in GitHub Actions and trigger it from there.

#!/bin/bash
#
APP_VALUE=$1

if [[ "$APP_VALUE" != "appOne" && "$APP_VALUE" != "appTwo" ]]; then
    echo "allowed values are appOne or appTwo"
    exit 1
fi

KVS_KEY="app-version"
KVS_ARN="arn:aws:cloudfront::123456789012:key-value-store/3add6be2-77d2-4d3a-9ddf-bc364519246e"
DIST_ID="EHVCAFT76II6T"

ETAG=$(aws cloudfront-keyvaluestore describe-key-value-store --kvs-arn "$KVS_ARN" --region us-east-1 --query "ETag" --output text)
aws cloudfront-keyvaluestore put-key --kvs-arn "$KVS_ARN" --if-match "$ETAG" --region us-east-1 --key "$KVS_KEY" --value "$APP_VALUE"

INVALIDATION_ID=$(aws cloudfront create-invalidation --distribution-id "$DIST_ID" --paths "/*" --region us-east-1 --query "Invalidation.Id" --output text)
echo "Invalidation ID: $INVALIDATION_ID"

aws cloudfront wait invalidation-completed --distribution-id "$DIST_ID" --id "$INVALIDATION_ID" --region us-east-1

echo "Invalidation completed"

I can switch app origin by running

./update_key.sh appOne

With this approach, I don't need to invalidate the cache, but I leave this step in the script anyway.

Summary

Code is available in this repository

CloudFront Functions are a convenient and effective way to implement conditional selection of origins. Using KeyValueStore allows receiving parameters without any network call, which keeps function execution immediate. Using custom headers passed to the origins, the cache is created separately for each one.

With a simple script, I can easily switch between origins.