DEV Community: Daniel Muller

Diving into Cloudfront

Daniel Muller — Fri, 17 Feb 2023 17:52:40 +0000

Cloudfront can be simply defined as a CDN (Content Delivery Network), caching your static assets in a datacenter nearer to your viewers. But Cloudfront is a lot more complex and versatile than this simple definition.

Cloudfront is a "pull" CDN, which means that you don't push your content to the CDN. The content is pulled into the CDN Edge from the origin at the first request of any piece of content.

In addition to the traditional pull and cache usage, Cloudfront can also be used as:

A Networking Router
A Firewall
A Web Server
An Application Server

Why is using a CDN relevant?

The main reason is to improve the speed of delivery of static content. By caching the content on the CDN edge, you not only reduce the download time from a few seconds to a few milliseconds, but you also reduce the load and amount of requests on your backend (Network, IO, CPU, Memory, ...).

Static content can be defined as content not changing between two identical requests done in the same time frame.

Identical can be as simple as the same URI, or as fine grained as down to the authentication header. The time frame can range between 1 second to 1 year.

The most common case, is caching resources like Javascript or CSS and serving the same file to all users for ever. But caching a JSON response tailored to a user (Authentication header) for a few seconds reduces the backend calls when the user has the well known "frenetic browser reload syndrome".

Edges, Mid-Tier Caches and Origins

Cloudfront isn't "just" some servers in datacenters around the world. The service is a layered network of Edge Locations and Regional Edge Caches (or Mid-Tier Caches).

Edge Locations are distributed around the globe with more than 400 points of presence over 90 cities across 48 countries. Each Edge Location is connected to one of the 13 Regional Edge Caches.

Regional Edge Caches are transparent to you and your visitors, you can't configure them or access them directly. Your visitors will interact with the nearest Edge Location, which will connect to the attached Regional Edge Cache and finally to your origin. Therefore, in this article, we will refer to Cloudfront as the combination of Edge Locations and Region Edge Caches.

Not only will the visitors benefit on download speed by retrieving content cached on the same Edge Location, but visitors in the same region using different Edge Locations will also benefit from the content cached at the Regional Edge Cache level by not having the need retrieve the content from the Origin.

Classic Pull and Cache Setup

Before diving into the deep end, let's refresh our memories with some of the basics of Cloudfront.

Cloudfront can allow write actions (POST, PUT, DELETE) but they will never be cached. We will mainly focus on the read actions (GET, HEAD, OPTIONS).

Origin types

HTTP/HTTPS
- S3 Website
- S3 Multi-Region Access Points
- API Gateway
- ALB
- Any HTTP Webserver
S3
- S3 API
Elemental Media Store Container
Elemental Media Package Container

S3 Website or S3 API ?

Even if they look very similar, this two ways of accessing S3 are very different.

S3 Website:
- The bucket needs to be configured with website capabilities
- Origin Domain: bucketName.s3-website-region.amazonaws.com
- Configured as CustomOrigin
- Pros:
- GET /foo/bar/ will serve /foo/bar/index.html
- Returns 404 on file not found (if listBucket is public)
- Cons:
- HTTP origin only
- Files in the bucket need to be public
- Files can be accessed by calling directly the S3-Website endpoint
S3 API:
- Origin Domain: bucketName.s3.amazonaws.com (or bucketName.s3.region.amazonaws.com)
- Configured as S3Origin
- Pros:
- HTTPS origin only
- Files can be private and Cloudfront is granted access using OAC
- Cons:
- GET /foo/bar/ returns 403 and not index.html

In both cases, we can overcome some of the cons:

S3 Website:
- Spoofing the referer header on the origin call and using an S3 policy to enforce this header minimizes the possibility to access the bucket directly

  ```json
  {
    "Version": "2012-10-17",
    "Id": "Cloudfront",
    "Statement": [
      {
        "Sid": "Cloudfront",
        "Effect": "Allow",
        "Principal": "*",
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::myBucket/*",
        "Condition": {
          "StringLike": {
            "aws:Referer": "AAAAAA"
          }
        }
      }
    ]
  }
  ```

S3 API:
- Using Lambda@Edge, we can rewrite the origin URI to serve index.html (more details below).

S3 Multi-Region Access Points

Sadly you won't see it as a possible origin, you need to treat it as a custom HTTPS endpoint.

To allow access to the origin you need to sign all the requests with AWS Sig4 by yourself using Lambda@Edge.

Custom HTTP(S) origins

For any HTTP/HTTPS origin, you can configure how Cloudfront connects to it:

Enforce HTTPS (or HTTP)
Custom ports
Timeouts
Additional Headers
...

Multiple Origins and Behaviors

Cloudfront can handle multiple origins. The routing is based on the request path and are configured as behaviors.

Each behavior has it's own settings of origin to use and how to handle cache policies.

It is not uncommon to have multiple behaviors sharing the same origin to adjust caching policies for different kind of content.

Custom Domain

Cloudfront will always attach their xxxxx.cloudfront.net domain to your distribution. But you have the possibility to use your own domain by defining one or more domain aliases.

SSL Termination

Cloudfront handles SSL terminations at the Edge. You can only attach a single SSL certificate. This certificate needs to list all domains used as aliases. Furthermore, your clients need to support SNI (all modern browser do), else you need to purchase dedicated IP addresses for your distribution.

You need to provision the certificate in us-east-1, which is an annoyance when working with Cloudformation and deploying to another region.

This allows to serve HTTPS traffic to your client, but connect using HTTP to the origin (S3 Website) or HTTPS on another domain (ALB, API Gateway).

Cache Policies and Cache Key

Caching is controlled using cache policies. Each behavior has a cache policy.

The cache policy not only dictates what makes a content vary, but also sets the time an object should remain in cache and if the response should be compressed or not. Values defined as Cache Key are automatically transferred to the Origin.

Cloudfront provides a set of pre-defined set of cache policies:

CachingOptimized: Ideal for S3 backends
- Headers, Cookies and Query String aren't taken into consideration
- Origin's Max-Age is used, defaults to 1 day
- Response is compressed
CachingDisabled:
- Headers, Cookies and Query String aren't taken into consideration
- Origin's Max-Age is overwritten to 0 (no cache)
- Response is compressed
MediaPackage: Ideal for MediaPackage origin
- origin Header is used to vary content
- Query String parameters aws.manifestfilter, start, end, m are used to vary content
- Cookies aren't taken into account
- Origin's Max-Age is used, defaults to 1 day
- Response is compressed

You can create your own depending on your needs, some examples:

Visitor's Language:
- Use the Browser's Header Accept-Language
Visitor's Country:
- Use the Header generated by Cloudfront based on the visitor's IP: CloudFront-Viewer-Country
API with pagination:
- Use the Query String Parameter page

Cache policies aren't linked to a specific distribution. You can re-use the same policy on different and unrelated distributions.

Max-Age and Cache-Control

Cloudfront will use the Origin's Max-Age (Cache-Control) if it is in the bounds of min TTL and max TTL.

If the Origin's TTL is outside the defined bounds, cloudfront will overwrite it with min TTL or max TTL.

If the Origin doesn't set any caching rule, the Default TTL value will be used.

Cloudfront's behavior on TTL doesn't affect the clients behavior. The Cache-Control header from the Origin is sent unmodified to the client.

Cloudfront might evict your object before the Max-Age is reached, specially if the object isn't requested often, but it won't never keep it longer than that value.

The browser will also cache for the amount of Max-Age, to cache the object differently in Cloudfront, you can use a combination of Max-Age and s-maxage. Cloudfront will cache the object for a duration of s-maxage and the browser for a duration of Max-Age. This is useful when using Cloudfront Functions to validate access token, when you want the browser to re-validate access but without having to fetch the object from the Origin.

Max-Age isn't the only way to control cache, Expires. For more in-depth details on the usage of Max-Age, refer to Cloudfront's expiration documentation.

Extending with edge functions

Cloudfront isn't limited at only a fetching and caching statically defined resources. It gives you the ability to modify requests and responses in-flight using functions deployed and executed at the edge.

You are able to interact with the request/response in four different steps of the execution.

viewer-request:
- Triggered before the request reaches Cloudfront
- Access to the first 40 KB of the body
- A modified body can have a maximum size of 53.2 KB
- Executed on each request
- Generate a response and by-pass the Origin Request
- Use cases:
- Normalize headers
- Normalize query string
- Normalize cache keys
- Authorization
origin-request:
- Triggered before Cloudfront calls the Origin
- Access to the first 1 MB of the body
- A modified body can have a maximum size of 1.33 MB
- Executed on cache miss only
- Generate a response and by-pass the Origin Request
- Use cases:
- Rewriting path or host
- Generate cached redirects
- Generate cached static responses
origin-response:
- Triggered before Cloudfront received the Origin response
- No access to the body
- Body can be generated
- Executed on cache miss only
- Use cases:
- Add headers
- Change HTTP status
- Replace body with static content
viewer-response:
- Triggered before Cloudfront sends the response to the client
- No access to the body
- Body can be generated
- Executed on each response
- Use cases:
- Add dynamic CORS headers

You have two solutions to interact with this requests/responses:

Cloudfront Functions:
- Executed on the Edge Location
- Access to viewer-request and viewer-response only
- Function can be deployed in any region. It's a Cloudfront Object
- Logs are stored in Cloudwatch Logs in us-east-1
- No access to body
- Geolocation Headers available
- Runtime: javascript (ES 5.1) with some restrictions:
- No custom ENV variables
- No module includes
- No timers
- No network access
- No File system access
- Crypto and QueryString modules are built-in
- Cost: $0.10 / 1M executions
- Use cases:
- Header normalization, manipulation
- Query String normalization
- Cache key generation
- Path rewrite
- Static Authorization
Lambda@Edge:
- Executed at the Regional Edge Cache
- Access to all 4 events
- Function needs to be deployed to us-east-1 as a Lambda Function
- Logs are stored in Cloudwatch Logs in the region of the Regional Edge Cache
- Limited access on request body
- 5s timeout on viewer request/response
- 30s timeout on origin request/response
- Geolocation Headers available
- Runtime: NodeJS or Python with some restrictions:
- No custom ENV variables
- No Lambda DLQ
- No VPC
- No Lambda Layers
- No Lambda Container Images
- Cost: $0.60 / 1M executions and additionally for the execution duration.
- Use cases:
- Authorization with oAuth endpoint
- Database query
- Body rewrite/validation

When to use which depends on the use case. If you don't need to access network resources, Cloudfront Functions is the right choice. Even if they are executed on each request (viewer-request), you would need a very good hit ratio to get cheaper executions with Lambda@Edge (when executed only on cache miss on origin-request).

Furthermore, the need to deploy Lambda@Edge to us-east-1 makes it a cumbersome task when you want to define your stack in a single Cloudformation template deployed to any other region than us-east-1.

If you need to access network resources or need more memory and time to execute your code, then Lambda@Edge is the right choice.

If neither of this solutions works for you, you still can deploy Lambda and access it via API Gateway or Lambda function URL deployed in one or multiple regions and cache the result in Cloudfront.

Event structure

Events for Lambda@Edge or Cloudfront Functions differ in some parts, mainly how headers are represented.

Viewer Request

Lambda@Edge

{
  "Records": [
    {
      "cf": {
        "config": {
          "distributionDomainName": "d111111abcdef8.cloudfront.net",
          "distributionId": "EDFDVBD6EXAMPLE",
          "eventType": "viewer-request",
          "requestId": "4TyzHTaYWb1GX1qTfsHhEqV6HUDd_BzoBZnwfnvQc_1oF26ClkoUSEQ=="
        },
        "request": {
          "clientIp": "203.0.113.178",
          "headers": {
            "host": [
              {
                "key": "Host",
                "value": "d111111abcdef8.cloudfront.net"
              }
            ],
            "user-agent": [
              {
                "key": "User-Agent",
                "value": "curl/7.66.0"
              }
            ],
            "accept": [
              {
                "key": "accept",
                "value": "*/*"
              }
            ]
          },
          "method": "GET",
          "querystring": "",
          "uri": "/"
        }
      }
    }
  ]
}

Cloudfront Function

{
    "version": "1.0",
    "context": {
        "distributionDomainName": "d111111abcdef8.cloudfront.net",
        "distributionId": "EDFDVBD6EXAMPLE",
        "eventType": "viewer-request",
        "requestId": "4TyzHTaYWb1GX1qTfsHhEqV6HUDd_BzoBZnwfnvQc_1oF26ClkoUSEQ=="
    },
    "viewer": {
        "ip": "203.0.113.178"
    },
    "request": {
        "method": "GET",
        "uri": "/",
        "querystring": {},
        "headers": {
            "host": {
                "value": "d111111abcdef8.cloudfront.net"
            },
            "user-agent": {
                "value": "curl/7.85.0"
            },
            "accept": {
                "value": "*/*"
            }
        },
        "cookies": {}
    }
}

Origin Request

Lambda@Edge

{
  "Records": [
    {
      "cf": {
        "config": {
          "distributionDomainName": "d111111abcdef8.cloudfront.net",
          "distributionId": "EDFDVBD6EXAMPLE",
          "eventType": "origin-request",
          "requestId": "4TyzHTaYWb1GX1qTfsHhEqV6HUDd_BzoBZnwfnvQc_1oF26ClkoUSEQ=="
        },
        "request": {
          "clientIp": "203.0.113.178",
          "headers": {
            "x-forwarded-for": [
              {
                "key": "X-Forwarded-For",
                "value": "203.0.113.178"
              }
            ],
            "user-agent": [
              {
                "key": "User-Agent",
                "value": "Amazon CloudFront"
              }
            ],
            "via": [
              {
                "key": "Via",
                "value": "2.0 2afae0d44e2540f472c0635ab62c232b.cloudfront.net (CloudFront)"
              }
            ],
            "host": [
              {
                "key": "Host",
                "value": "example.org"
              }
            ],
            "cache-control": [
              {
                "key": "Cache-Control",
                "value": "no-cache, cf-no-cache"
              }
            ]
          },
          "method": "GET",
          "origin": {
            "custom": {
              "customHeaders": {},
              "domainName": "example.org",
              "keepaliveTimeout": 5,
              "path": "",
              "port": 443,
              "protocol": "https",
              "readTimeout": 30,
              "sslProtocols": [
                "TLSv1",
                "TLSv1.1",
                "TLSv1.2"
              ]
            }
          },
          "querystring": "",
          "uri": "/"
        }
      }
    }
  ]
}

Origin Response

Lambda@Edge

{
  "Records": [
    {
      "cf": {
        "config": {
          "distributionDomainName": "d111111abcdef8.cloudfront.net",
          "distributionId": "EDFDVBD6EXAMPLE",
          "eventType": "origin-response",
          "requestId": "4TyzHTaYWb1GX1qTfsHhEqV6HUDd_BzoBZnwfnvQc_1oF26ClkoUSEQ=="
        },
        "request": {
          "clientIp": "203.0.113.178",
          "headers": {
            "x-forwarded-for": [
              {
                "key": "X-Forwarded-For",
                "value": "203.0.113.178"
              }
            ],
            "user-agent": [
              {
                "key": "User-Agent",
                "value": "Amazon CloudFront"
              }
            ],
            "via": [
              {
                "key": "Via",
                "value": "2.0 8f22423015641505b8c857a37450d6c0.cloudfront.net (CloudFront)"
              }
            ],
            "host": [
              {
                "key": "Host",
                "value": "example.org"
              }
            ],
            "cache-control": [
              {
                "key": "Cache-Control",
                "value": "no-cache, cf-no-cache"
              }
            ]
          },
          "method": "GET",
          "origin": {
            "custom": {
              "customHeaders": {},
              "domainName": "example.org",
              "keepaliveTimeout": 5,
              "path": "",
              "port": 443,
              "protocol": "https",
              "readTimeout": 30,
              "sslProtocols": [
                "TLSv1",
                "TLSv1.1",
                "TLSv1.2"
              ]
            }
          },
          "querystring": "",
          "uri": "/"
        },
        "response": {
          "headers": {
            "access-control-allow-credentials": [
              {
                "key": "Access-Control-Allow-Credentials",
                "value": "true"
              }
            ],
            "access-control-allow-origin": [
              {
                "key": "Access-Control-Allow-Origin",
                "value": "*"
              }
            ],
            "date": [
              {
                "key": "Date",
                "value": "Mon, 13 Jan 2020 20:12:38 GMT"
              }
            ],
            "referrer-policy": [
              {
                "key": "Referrer-Policy",
                "value": "no-referrer-when-downgrade"
              }
            ],
            "server": [
              {
                "key": "Server",
                "value": "ExampleCustomOriginServer"
              }
            ],
            "x-content-type-options": [
              {
                "key": "X-Content-Type-Options",
                "value": "nosniff"
              }
            ],
            "x-frame-options": [
              {
                "key": "X-Frame-Options",
                "value": "DENY"
              }
            ],
            "x-xss-protection": [
              {
                "key": "X-XSS-Protection",
                "value": "1; mode=block"
              }
            ],
            "content-type": [
              {
                "key": "Content-Type",
                "value": "text/html; charset=utf-8"
              }
            ],
            "content-length": [
              {
                "key": "Content-Length",
                "value": "9593"
              }
            ]
          },
          "status": "200",
          "statusDescription": "OK"
        }
      }
    }
  ]
}

Viewer Response

Lambda@Edge

{
  "Records": [
    {
      "cf": {
        "config": {
          "distributionDomainName": "d111111abcdef8.cloudfront.net",
          "distributionId": "EDFDVBD6EXAMPLE",
          "eventType": "viewer-response",
          "requestId": "4TyzHTaYWb1GX1qTfsHhEqV6HUDd_BzoBZnwfnvQc_1oF26ClkoUSEQ=="
        },
        "request": {
          "clientIp": "203.0.113.178",
          "headers": {
            "host": [
              {
                "key": "Host",
                "value": "d111111abcdef8.cloudfront.net"
              }
            ],
            "user-agent": [
              {
                "key": "User-Agent",
                "value": "curl/7.66.0"
              }
            ],
            "accept": [
              {
                "key": "accept",
                "value": "*/*"
              }
            ]
          },
          "method": "GET",
          "querystring": "",
          "uri": "/"
        },
        "response": {
          "headers": {
            "access-control-allow-credentials": [
              {
                "key": "Access-Control-Allow-Credentials",
                "value": "true"
              }
            ],
            "access-control-allow-origin": [
              {
                "key": "Access-Control-Allow-Origin",
                "value": "*"
              }
            ],
            "date": [
              {
                "key": "Date",
                "value": "Mon, 13 Jan 2020 20:14:56 GMT"
              }
            ],
            "referrer-policy": [
              {
                "key": "Referrer-Policy",
                "value": "no-referrer-when-downgrade"
              }
            ],
            "server": [
              {
                "key": "Server",
                "value": "ExampleCustomOriginServer"
              }
            ],
            "x-content-type-options": [
              {
                "key": "X-Content-Type-Options",
                "value": "nosniff"
              }
            ],
            "x-frame-options": [
              {
                "key": "X-Frame-Options",
                "value": "DENY"
              }
            ],
            "x-xss-protection": [
              {
                "key": "X-XSS-Protection",
                "value": "1; mode=block"
              }
            ],
            "age": [
              {
                "key": "Age",
                "value": "2402"
              }
            ],
            "content-type": [
              {
                "key": "Content-Type",
                "value": "text/html; charset=utf-8"
              }
            ],
            "content-length": [
              {
                "key": "Content-Length",
                "value": "9593"
              }
            ]
          },
          "status": "200",
          "statusDescription": "OK"
        }
      }
    }
  ]
}

Cloudfront Function

{
    "version": "1.0",
    "context": {
        "distributionDomainName": "d111111abcdef8.cloudfront.net",
        "distributionId": "EDFDVBD6EXAMPLE",
        "eventType": "viewer-response",
        "requestId": "MmxS5hbDhc9VyOIqzmYksKesOj6n_54ycCBX4XCS5-w7OJJ5wloOAA=="
    },
    "viewer": {
        "ip": "203.0.113.178"
    },
    "request": {
        "method": "GET",
        "uri": "/",
        "querystring": {},
        "headers": {
            "host": {
                "value": "d111111abcdef8.cloudfront.net"
            },
            "user-agent": {
                "value": "curl/7.85.0"
            },
            "accept": {
                "value": "*/*"
            }
        },
        "cookies": {}
    },
    "response": {
        "statusCode": 200,
        "statusDescription": "OK",
        "headers": {
            "date": {
                "value": "Fri, 25 Nov 2022 12:33:42 GMT"
            },
            "last-modified": {
                "value": "Fri, 25 Nov 2022 12:31:12 GMT"
            },
            "etag": {
                "value": "\"b9c2e628c3ffe65db36c4d92c9aebbb3\""
            },
            "accept-ranges": {
                "value": "bytes"
            },
            "server": {
                "value": "AmazonS3"
            },
            "via": {
                "value": "1.1 dfeaaa9951aa7df30bdb3dfb8a94470a.cloudfront.net (CloudFront)"
            },
            "age": {
                "value": "82"
            },
            "content-type": {
                "value": "text/html"
            },
            "content-length": {
                "value": "109"
            }
        },
        "cookies": {}
    }
}

Less conventional usages

Networking Router

Even if you don't need the caching functionalities of Cloudfront (POST requests or disabling cache on GET), you can still use Cloudfront to act as a Networking Router, by sending your visitor traffic to the nearest edge. From the edge to the origin, the performant AWS Backbone will be used instead of your ISP's peering.

By using multiples behaviors, you can route your traffic to different backends that don't need to be in the same region or even inside AWS.

As an example, S3 Transfer Acceleration uses Cloudfront as the endpoint, forcing the networking path to the nearest Edge Location and leveraging the AWS backbone to reach the bucket.

Firewall

In addition to use AWS WAF with Cloudfront to protect your Origin application, Cloudfront also provides a default DDOS protection. You can also deny access to visitors from specific countries.

By adding Origin Shield, you add an additional layer of caching between Cloudfront and your Origin, reducing the need for Cloudfront to retrieve content from your Origin. This helps to reduce the load on your origin and improves the CDN hit ratio (and therefore download speed) for your visitors.

Application Server

By using Lambda@Edge functions, you can directly query databases like DynamoDB and apply business logic to it, without needing any Origin (you still need to configure a dummy one, Cloudfront doesn't allow you to be Origin-less).

Combining this with DynamoDB Global Tables, you can always query a table near your edge, making your application performant and reliable.

Web Server

As seen through all the examples mentioned in this article, Cloudfront can be seen as "just" an HTTP server in front of your application.

Using functions (or Lambda@Edge) you can return redirections or static content without the need of any backend.

By using multiples behaviors, you can route your traffic to different types backends.

Real World Use Cases

You can find the template used for the below examples on Github.

The template provides a distribution with different backends:

S3 Website
- /blog/*
- /private/*
S3 API
- /assets/*
- /html/*
- /airport/*
API Gateway
- /*

Cache based on visitor's country

Backend: Api Gateway + Lambda
Edge function: None
Policy: Whitelist cloudfront-viewer-country

  Type: AWS::CloudFront::CachePolicy
  Properties:
    CachePolicyConfig:
      DefaultTTL: 10
      MinTTL: 0
      MaxTTL: 3600
      Name: Country
      ParametersInCacheKeyAndForwardedToOrigin:
        CookiesConfig:
          CookieBehavior: none
        EnableAcceptEncodingBrotli: true
        EnableAcceptEncodingGzip: true
        HeadersConfig:
          HeaderBehavior: whitelist
          Headers:
            - cloudfront-viewer-country
        QueryStringsConfig:
          QueryStringBehavior: none

Request:

curl -v 'https://d3h57w0cnyb350.cloudfront.net/country'

> GET /country HTTP/2
> Host: d3h57w0cnyb350.cloudfront.net
> user-agent: curl/7.85.0
> accept: */*

< HTTP/2 200 
< content-type: application/json
< content-length: 2
< date: Tue, 06 Dec 2022 17:58:23 GMT
< x-amz-cf-pop: LIS50-C1

Lambda Event:

{
    "version": "2.0",
    "routeKey": "GET /country",
    "rawPath": "/country",
    "rawQueryString": "",
    "headers": {
        "accept-encoding": "br,gzip",
        "cloudfront-viewer-country": "PT",
        "content-length": "0",
        "host": "wvo7t33pz3.execute-api.eu-central-1.amazonaws.com",
        "user-agent": "Amazon CloudFront",
        "via": "2.0 592fdb72142153f4ac204b48e22d9036.cloudfront.net (CloudFront)",
        "x-amz-cf-id": "kz-vrWiS6p5lg9ZjGRCY7Xuwg2gmr5utkrJLaU62Leol8ApRIzL4nw==",
        "x-amzn-trace-id": "Root=1-638f8250-33b4ecae5ea02e382b6d5d8b",
        "x-forwarded-for": "X.X.X.X",
        "x-forwarded-port": "443",
        "x-forwarded-proto": "https"
    },
    "requestContext": {
        "accountId": "688589788262",
        "apiId": "wvo7t33pz3",
        "domainName": "wvo7t33pz3.execute-api.eu-central-1.amazonaws.com",
        "domainPrefix": "wvo7t33pz3",
        "http": {
            "method": "GET",
            "path": "/country",
            "protocol": "HTTP/1.1",
            "sourceIp": "X.X.X.X",
            "userAgent": "Amazon CloudFront"
        },
        "requestId": "cvFMihZBliAEPSw=",
        "routeKey": "GET /country",
        "stage": "$default",
        "time": "06/Dec/2022:17:56:32 +0000",
        "timeEpoch": 1670349392038
    },
    "isBase64Encoded": false
}

Rewrite URI to serve index.html

Nobody wants to type /index.html when calling a URL. Apache, Nginx and S3-Website are loading index.html automatically when no document is passed in the URI. When using an S3-API backend we need to provide this functionality ourselves. Cloudfront is only able to load index.html at the root, which is generally enough for an SPA but not for a static generated site.

S3-Website

Backend: S3-Website
Edge function: None
Policy: managed-cacheOptimized

Request:

curl -v 'https://d3h57w0cnyb350.cloudfront.net/blog/articles/'

< HTTP/2 200 
< content-type: text/html

index.html is returned, the backend is an S3-Website and has the logic to serve index.html when document name is provided.

S3-API without function

Backend: S3-API
Edge function: None
Policy: managed-cacheOptimized

Request:

curl -v 'https://d3h57w0cnyb350.cloudfront.net/assets/articles/'

< HTTP/2 403
< content-type: application/xml

An error is returned. The document /assets/articles/ doesn't exist in S3.

S3-API with function

Backend: S3-API
Edge function: viewer-request Cloudfront Function
Policy: managed-cacheOptimized

function:

function handler(event) {
  var request = event.request
  var uri = request.uri
  if (uri.endsWith('/')) {
    request.uri += 'index.html';
  }
  else if (!uri.includes('.')) {
    request.uri += '/index.html';
  }
  return request;
}

Request:

curl -v 'https://d3h57w0cnyb350.cloudfront.net/html/articles/'

< HTTP/2 200
< content-type: text/html

index.html is returned, we rewrite the incoming URI to append index.html to the request so that an existing key can be fetched from S3.

Serve localized content

Using the browser's accept-language header, we are returning the content in the language requested by the viewer. The URL is the same regardless of the language, but needs to be cached according to this language. Since this header can have multiple variations, we normalize it to increase our hit ratio.

Backend: S3-API
Edge function: viewer-request Cloudfront Function
Policy: Whitelist x-locale

function:

function handler(event) {
  /*
  * Rewrite URI to serve index.html when missing
  **/
  var request = event.request

  if (request.uri.endsWith('/')) {
    request.uri += 'index.html';
  }
  else if (!request.uri.includes('.')) {
    request.uri += '/index.html';
  }

  /*
  * Set default locale
  **/
  var locale = 'en'
  var translations = ['fr', 'de']

  /*
  * Parse accept-language and extract the first value for simplicity
  * We should use all languages and use the first for which we have a translation
  **/
  if (request.headers['accept-language'] && request.headers['accept-language'].value) {
    var language = request.headers['accept-language'].value.split(',')[0].split(';')[0].substring(0,2).toLowerCase()
    if (translations.indexOf(language) > -1) {
      locale = language
    }
  }

  /*
  * Create an x-locale header to be used as part of the cache key
  **/
  request.headers['x-locale'] = {
    value: locale
  }
  /**
   * Rewrite the URI
   */
  request.uri = request.uri.replace('locale/', `locale/${locale}/`)
  return request;
}

Default (English):

curl https://d3h57w0cnyb350.cloudfront.net/html/locale/

< x-cache: Miss from cloudfront

<html>
  <head>
    <title>English</title>
  </head>
  <body>
    <h1>English content</h1>
  </body>
</html>

French:

curl -H "Accept-Language: fr" https://d3h57w0cnyb350.cloudfront.net/html/locale/

< x-cache: Miss from cloudfront

<html>
  <head>
    <title>French</title>
  </head>
  <body>
    <h1>French content</h1>
  </body>
</html>

Swiss-French:

curl -H "Accept-Language: fr-CH" https://d3h57w0cnyb350.cloudfront.net/html/locale/

< x-cache: Hit from cloudfront

<html>
  <head>
    <title>French</title>
  </head>
  <body>
    <h1>French content</h1>
  </body>
</html>

Even with a different Accept-Language header, the call for fr-CH is a Hit. Both languages are normalized to fr.

Protect with password

To showcase how password protection work, we will use "Basic Authentication". In a more realistic setup, we would use a JWT token validation. In both cases, the authorization is stored inside the function's code. If this is a security concern, you would need to use a Lambda@Edge function and validate the token by calling the Authorization service.

Backend: S3-Website
Edge function: viewer-request Cloudfront Function
Policy: managed-cacheOptimized

function:

function handler(event) {
  var authHeaders = event.request.headers.authorization
  /**
   * Authorization string is sent by the browser as base64Encode(username:password).
   * base64Encode('private:private')='cHJpdmF0ZTpwcml2YXRl'
   */
  var expected = "Basic cHJpdmF0ZTpwcml2YXRl"
  if (authHeaders && authHeaders.value === expected) {
    return event.request
  }

  var response = {
    statusCode: 401,
    statusDescription: "Unauthorized",
    headers: {
      "www-authenticate": {
        value: "Basic realm='Enter your credentials'"
      }
    }
  }
  return response
}

With a valid authentication, the request is returned, instructing Cloudfront to continue by sending the request to the Origin.

Without any valid authentication, a response is returned, instructing Cloudfront to not call the Origin and directly return the response to the client.

curl -v https://d3h57w0cnyb350.cloudfront.net/private/

< HTTP/2 401 
< www-authenticate: Basic realm='Enter your credentials'
< x-cache: FunctionGeneratedResponse from cloudfront

curl -v https://private:private@d3h57w0cnyb350.cloudfront.net/private/

> authorization: Basic cHJpdmF0ZTpwcml2YXRl

< HTTP/2 200 
< content-type: text/html

<html>
  <head>
    <title>Private</title>
  </head>
  <body>
    <h1>Private Content</h1>
  </body>
</html>

Fetch content from DynamoDB

We use a Lambda@Edge function on the origin-request event. We fetch data from a DynamoDB Table and return the result, bypassing the request to the Origin.

By using origin-request, we can cache the response in Cloudfront and trigger the function only when the result isn't already cached in Cloudfront.

Backend: Any (not used but must be provided)
Edge function: origin-request Lambda@Edge
Policy: managed-cacheOptimized

function:

import type { CloudFrontRequestEvent, CloudFrontResponseResult } from 'aws-lambda'
import aws from 'aws-sdk'
import https from 'https'

const documentClient = new aws.DynamoDB.DocumentClient({
  region: 'eu-central-1',
  httpOptions: {
    agent: new https.Agent({
      keepAlive: true,
    }),
  },
})

export const handler = async (event: CloudFrontRequestEvent): Promise<CloudFrontResponseResult> => {
  const request = event.Records[0].cf.request

  try {
    const code = request.uri.split('/').slice(-1)[0]

    const data = await documentClient
      .get({
        TableName: 'demoContent',
        Key: {
          code,
        },
      })
      .promise()
    if (!(data && data.Item && data.Item.name)) {
      return notFound
    }
    return {
      status: '200',
      statusDescription: 'OK',
      body: JSON.stringify(data.Item),
      headers: {
        'content-type': [
          {
            value: 'application/json',
          },
        ],
        'cache-control': [
          {
            value: 'max-age=120',
          },
        ],
      },
    }
  } catch () {
    return notFound
  }
}

const notFound = {
  status: '404',
  statusDescription: 'Not Found',
  headers: {
    'content-type': [
      {
        value: 'application/json',
      },
    ],
    'cache-control': [
      {
        value: 'max-age=10',
      },
    ],
  },
  body: JSON.stringify({ message: 'Object Not Found' }),
}

curl -v "https://d3h57w0cnyb350.cloudfront.net/airport/gva"

< x-cache: Miss from cloudfront

curl -v "https://d3h57w0cnyb350.cloudfront.net/airport/gva"

< x-cache: Hit from cloudfront

{
  "city":"Geneva",
  "code":"gva",
  "name":"Geneva International Airport",
  "country":"Switzerland"
}

Serve alternate content in case of error

When the Origin is unreachable, overloaded or simply unable to handle the request, instead of showing the error to the client we redirect him to an alternate site.

Backend: Any
Edge function: origin-response Lambda@Edge
Policy: managed-cacheOptimized

function:

import type { CloudFrontResponseEvent, CloudFrontResponseResult } from 'aws-lambda'

export const handler = async (
  event: CloudFrontResponseEvent,
): Promise<CloudFrontResponseResult> => {
  const response = event.Records[0].cf.response
  const request = event.Records[0].cf.request

  try {
    const status = parseInt(response.status)
    if (status >= 400 && status <= 599) {
      return redirect(request.uri)
    } else {
      return response
    }
  } catch (e) {
    // eslint-disable-next-line no-console
    console.error(e as Error)

    return redirect(request.uri)
  }
}

const redirect = (uri: string): CloudFrontResponseResult => {
  return {
    status: '307',
    statusDescription: 'Temporary Redirect',
    headers: {
      location: [
        {
          value: `https://alt.example.com${uri}`,
        },
      ],
    },
  }
}

What have we learnt?

Cloudfront is more than just a simple "pull-cache-serve" service
You improve delivery speed to your visitors
You can increase resilience by always using a healthy backend
You improve overall speed to your backend by leveraging AWS's backbone
You can modify any request to tailor the response to your visitor's device or region
You don't always need a backend
You protect your backend by reducing the amount of calls reaching it

Convert and distribute your videos with AWS Elemental

Daniel Muller — Wed, 01 Feb 2023 09:52:19 +0000

You are in possession of some videos that you want to distribute or share.

Your use case could be as simple as needing to share a lower resolution of a video taken on your smartphone or camera. Or your use case could be as complex as sharing production-quality copies of the latest movie you are producing.

For both cases you can use Serverless Cloud Products to address your needs. AWS Elemental MediaConvert, one of the Media tools from the AWS Elemental family, will allow you to solve your problem.

Or perhaps, you want to distribute your content to viewers around the globe allowing them to watch on any device? Then you can use another Serverless Cloud Product from the AWS Elemental family: AWS Elemental MediaPackage.

Services used

Video files are complex. You need to worry about image and audio, which means you need to worry about container, codecs, bitrate, pixel aspect ratio and more.

MediaConvert

MediaConvert allows you to transcode file-based content. This means you can transform a video file into a different format and size.

But MediaConvert is more that just transformation. We won't get into details of all functionalities in this article, but here are some noticeable features:

Watermarking
Graphic overlay (static or motion)
Select parts (time or size) of an Input
Rotation
Deinterlacing
...

MediaPackage

AWS Elemental MediaPackage is a Just-In-Time (JIT) media packager for your existing assets. It will generate the relevant manifest for a group of video sources.

It not only allows you to define multiple qualities of the same video, but allows to add multiple audio sources and different video sources like camera angles.

For our use case, we are only interested in providing different qualities of the same content and only a single audio track.

MediaPackage uses the formats created by MediaConvert and generates on the fly a manifest either in HLS or in Dash to be consumed by the player.

Simple Use case: Reduce and Convert for mobile sharing

You created a FHD (1920x1080) video with your camera. You camera creates movies in uncompressed QuickTime format. Most of the devices won't be able to read this format unless they have the right codecs installed.

To allow recipients to play your video, you need to convert it to an MP4 container and the H.264 codec (de facto standard for Web distribution). You will also want to improve the download speed.

Upload your source file to S3
Trigger a MediaConvert job using this file
- Convert to an MP4 Container
- Convert to H.264 codec
- Reduce the size to 1280x720

Result

	Source file	Converted file	Converted file
Container	QuickTime	MP4	MP4
Codec	H.264	H.264	H.264
Dimensions	1920x1080	1920x1080	1280x720
Duration	28s	28s	28s
Size	37 MB	21.7 MB	12.9 MB
Bitrate	10836 kb/s	6429 kb/s	3835 kb/s

Both outputs were created in a single MediaConvert job that took 17 seconds.

More complex use case: Create content for web distribution

In this use case, we want to distribute a 4K (3840x2160) 90 minutes movie. We want our viewer to enjoy our content on any type of screen: from small smartphones to big TV screens.

We also want them to enjoy it regardless of their network connectivity.

We need to be bandwidth conscious and not ship more than the viewer can consume.

MP4 container and H.264 Codec is a combination that can be viewed by most media players (smartphones, TV, Set-top boxes, Gaming devices, ...). We will use this format to distribute our content.

Screen size consideration

We pre-render our content to adapt to our viewers screen size. Reducing the dimensions also allows us to reduce the bandwith needed.

Unless mentioned, we will use H.264 and the source frame rate.

SD: 480 x 270, 400kbps, 15fps
SD: 640 x 360, 700kbps
SD: 854 x 480, 1Mbps
HD: 1280 x 720, 3.5Mbps
FHD: 1920 x 1080, 6Mbps
4K: 3840 x 2160, 20Mbps, H.265

Segmented videos: HLS and Dash

If we distribute a single file, the client would be stuck on a single quality. Without tweaks on the player side, the client would download the whole file, even the parts that aren't watched.

To allow a streaming-like experience, we will use a distribution format named HLS (HTTP Live Streaming) and Dash.

Both format are very similar in the concept, but the choice of their usage depends on the players OS. For simplification, one could say that HLS is for Apple devices and Dash for others, but in reality it's slightly more complex.

The idea behind theses formats, is to split the movie in little chunks of a few seconds and define the structure through a manifest file.

By doing this, the player will download little chunks of content in the dimensions appropriate for the screen and the available bandwidth. By downloading only a few segments ahead of the current timestamp, it will use only the bandwidth needed for what is really watched and adapt playback on the current conditions of the player (phone rotated, window resized, bandwidth alterations, ...).

By leveraging HTTP as a delivery mechanism, we not only rely on an universally approved protocol but also allow caching at the CDN level, improving distribution around the globe.

HLS manifests and video files

Generate HLS and Dash Segments

MediaConvert can generate segmented videos and store them to S3. But with everything generated statically once, you loose flexibility. There a multiple reasons you want to have a more dynamic way to generate your manifests:

Removing some renditions on client attributes: paid tiers, legal constraints in some countries
Ordering of renditions: Improve start time for some players
Cost of storing never accessed renditions
DRM protection
Additional audio or video tracks

The solution

Serverless stack: Build renditions and distribution of large video files

Configuration

MediaConvert outputs

Audio: AAC, 160kbps, 48kHz

"AudioDescriptions": [
    {
        "AudioSourceName": "Audio Selector 1",
        "AudioType": 0,
        "AudioTypeControl": "FOLLOW_INPUT",
        "Codec": "AAC",
            "CodecSettings": {
                "AacSettings": {
                "AudioDescriptionBroadcasterMix": "NORMAL",
                "Bitrate": 160000,
                "CodecProfile": "LC",
                "CodingMode": "CODING_MODE_2_0",
                "RateControlMode": "CBR",
                "RawFormat": "NONE",
                "SampleRate": 48000,
                "Specification": "MPEG4"
            }
        },
        "LanguageCodeControl": "FOLLOW_INPUT"
    }
]

Video: MP4, H.264/H.265, Quality based bitrate

Bitrate and dimensions are replaced for each output quality.

The GOPSize of 2 seconds is important, this allows to "cut" the MP4 on key frames every multiple of 2 seconds, providing a fast and safe way to generate segments from the source file.

{
    "ContainerSettings": {
        "Container": "MP4",
        "Mp4Settings": {
            "CslgAtom": "INCLUDE",
            "FreeSpaceBox": "EXCLUDE",
            "MoovPlacement": "PROGRESSIVE_DOWNLOAD",
        },
    },
    "VideoDescription": {
        "AfdSignaling": "NONE",
        "AntiAlias": "ENABLED",
        "Height": 1080,
        "Width": 1920,
        "CodecSettings": {
            "Codec": "H_264",
            "H264Settings": {
                "AdaptiveQuantization": "HIGH",
                "CodecLevel": "LEVEL_4_2",
                "CodecProfile": "HIGH",
                "EntropyEncoding": "CABAC",
                "FieldEncoding": "PAFF",
                "FlickerAdaptiveQuantization": "ENABLED",
                "FramerateControl": "SPECIFIED",
                "FramerateConversionAlgorithm": "DUPLICATE_DROP",
                "FramerateDenominator": 1001,
                "FramerateNumerator": 30000,
                "GopBReference": "DISABLED",
                "GopClosedCadence": 1,
                "GopSize": 2,
                "GopSizeUnits": "SECONDS",
                "HrdBufferInitialFillPercentage": 90,
                "HrdBufferSize": 12000000,
                "InterlaceMode": "PROGRESSIVE",
                "MaxBitrate": 6000000,
                "MinIInterval": 0,
                "NumberBFramesBetweenReferenceFrames": 1,
                "NumberReferenceFrames": 3,
                "ParControl": "SPECIFIED",
                "ParDenominator": 1,
                "ParNumerator": 1,
                "QualityTuningLevel": "SINGLE_PASS_HQ",
                "QvbrSettings": {
                    "QvbrQualityLevel": 8
                },
                "RateControlMode": "QVBR",
                "RepeatPps": "DISABLED",
                "SceneChangeDetect": "ENABLED",
                "Slices": 1,
                "SlowPal": "DISABLED",
                "Softness": 0,
                "SpatialAdaptiveQuantization": "ENABLED",
                "Syntax": "DEFAULT",
                "Telecine": "NONE",
                "TemporalAdaptiveQuantization": "ENABLED",
                "UnregisteredSeiTimecode": "DISABLED"
            }
        },
        "ColorMetadata": "INSERT",
        "DropFrameTimecode": "ENABLED",
        "RespondToAfd": "NONE",
        "ScalingBehavior": "STRETCH_TO_OUTPUT",
        "Sharpness": 50,
        "TimecodeInsertion": "DISABLED"
    }
}

[
  {
    "Width": 3840,
    "Height": 2160,
    "Bitrate": 20000000,
    "Profile": "MAIN-MAIN",
    "Level": "AUTO",
    "Codec": "H_265",
  },
  {
    "Width": 1920,
    "Height": 1080,
    "Bitrate": 6000000,
    "Profile": "HIGH",
    "Level": "LEVEL_4_2",
  },
  {
    "Width": 1280,
    "Height": 720,
    "Bitrate": 3500000,
    "Profile": "HIGH",
    "Level": "LEVEL_4_2",
  },
  {
    "Width": 854,
    "Height": 480,
    "Bitrate": 1000000,
    "Profile": "MAIN",
    "Level": "LEVEL_3_1",
  },
  {
    "Width": 640,
    "Height": 360,
    "Bitrate": 700000,
    "Profile": "MAIN",
    "Level": "LEVEL_3_1",
  },
  {
    "Width": 480,
    "Height": 270,
    "Bitrate": 400000,
    "FramerateNumerator": 15000,
    "Profile": "MAIN",
    "Level": "LEVEL_3_1",
  },
]

MediaPackage packaging groups

HLS
For HLS, we create segments of 6 seconds (industry standard)

Type: AWS::MediaPackage::PackagingConfiguration
Properties:
  Id: hls
  PackagingGroupId: !Ref MediaPackagePackagingGroup
  HlsPackage:
    HlsManifests:
      - ManifestName: index
        IncludeIFrameOnlyStream: true
        StreamSelection:
          StreamOrder: VIDEO_BITRATE_DESCENDING
    SegmentDurationSeconds: 6
    UseAudioRenditionGroup: false

Dash
For HLS, we create segments of 2 seconds (industry standard)

Type: AWS::MediaPackage::PackagingConfiguration
Properties:
  Id: dash
  PackagingGroupId: !Ref MediaPackagePackagingGroup
  DashPackage:
    DashManifests:
      - ManifestName: index
        ManifestLayout: FULL
        MinBufferTimeSeconds: 30
        StreamSelection:
          StreamOrder: VIDEO_BITRATE_DESCENDING
    SegmentDurationSeconds: 2
    SegmentTemplateFormat: NUMBER_WITH_TIMELINE

Incoming Bucket

Source files are uploaded to S3
A rule on EventBridge listens to "Object:Created" events and triggers the execution of the "Conversion" Step Function.

Conversion Step Function

Using a Lambda function and https://ffmpeg.org/, the source file is analyzed with ffprobe
- Bitrate
- Dimensions
- Codecs
The file name is parsed to provide a movie name
- A naming convention could be used to interact with IMDB to gather more informations
An entry is created in DynamoDB
The video is sent to Rekognition to extract labels, persons, ... Not implemented for this article
A Lambda function builds the outputs based on the source file definition and triggers MediaConvert
- We don't create a rendition bigger than the source
- We don't create renditions with a higher bitrate than the source
- Generate video stills to be used as covers
An EventBridge rule listens to executed MediaConvert jobs and triggers

Packaging Step Function

Store still informations to DynamoDB
- Images can be used by a CRM as Video covers
Store renditions informations to DynamoDB
- These files can be used for download (offline viewing)
A Lambda function creates a SMIL manifest
A Lambda function creates a package using the manifest and the pre-defined HLS and Dash outputs
The URLs are stored to DynamoDB

Consumption

The video metadata and sources can be provided to the client via an API (not implemented for this article).
The client accessed the video content directly from the HLS or Dash URL served via Cloudfront

Result in action

To showcase the solution, we used a simple source file shot on a Smartphone: 11 seconds FHD (1920x1080) of 25MB.

All we needed to do to produce ready consumable content, was to upload this file to S3. Our Serverless solution took care of all the underlying steps.

MP4

Source	11 s	1920 x 1080	25 MB	19.1 Mbps
1080p	11 s	1920 x 1080	8.6 MB	19.1 Mbps
720p	11 s	1280 x 720	5.0 MB	19.1 Mbps
480p	11 s	854 x 480	1.6 MB	19.1 Mbps
360p	11 s	640 x 360	7.3 MB	19.1 Mbps
270p	11 s	480 x 270	0.8 MB	19.1 Mbps

360p

Download MP4 video.

Stills

Adaptive bitrates

HLS

If your browser doesn't support HLS natively, you can access it on the hls-js demo page.

Dash

If your browser doesn't support Dash natively, you can access it on the dashif demo page.

What did we achieve?

We created a fully Serverless pipeline to transform a source video provided in any format in a group of ready consumable formats by any smartphone or browser anywhere in the world. Even viewers with limited bandwidth can enjoy our content without buffering, they will however settle for a lower quality.

By leveraging Serverless solutions, we get all the known benefits of serverless:

Cost is kept to a minimum by paying only for what we use
- Video Conversion: Pay per movie conversion
- Video Packaging: Pay per movie consumption
- CDN: Pay per movie consumption
- S3 Storage: Pay for stored content
- DynamoDB: Pay for stored data
No servers to provision or maintain
The services scales with usage automatically

Extending the solution

With the solution we built, we barely scratched the surface of what can be done. There are several addons that can be built on top of this solution:

Provide DRM to protect your content
Use multiple manifests to allow high definition to selected viewers
Monetize your content with in-stream ads by leveraging AWS Media Tailor
Create live feeds from your VOD assets
...

Use IAM Identity Center (AWS SSO) to protect your Cloudfront served application

Daniel Muller — Tue, 04 Oct 2022 22:45:18 +0000

Using Cloudfront to distribute your web application is a good practice, by leveraging caching at the Edge, you ensure that your visitors will get the lowest latency possible.

But what if you had a pre-release of your application that isn't for the whole world to see, but you still want to provide the same experience as you will in production? Cloudfront doesn't propose a solution out-of-the-box for this.

Lambda@Edge or Cloudfront-Functions

Luckily, Cloudfront has ways to act on request by using either Lambda@Edge or Cloudfront-Functions.

Several solutions already exist leveraging theses services to add different types of password protection or user authentication on Cloudfront:

HTTP Basic Authentication using Cloudfront-Functions: joshualyman.com
CloudFront authorization@edge: aws-samples

Using SAML Authentication with existing IAM Identity Center

You might have set-up your AWS Accounts using Control Tower with Organizations and are managing your members using IAM Identity Center (Successor to AWS Single-Sign-On). Or you are using AWS Identity Center as a standalone tool to centralize your SSO credentials for 3rd party applications.

The solution

The application is the SAML Service Provider (SP), IAM Identity Center is the SAML Identity Provider (IdP).

For each request, the SP validates the encrypted authorization cookie using a Lambda@Edge function in the viewer-request event.
If the Cookie isn't valid:
1. The Lambda@Edge function will create an Authorization Request and redirect the browser to the relevant IdP Endpoint.
2. The IdP provides the user with a login page.
3. Upon successful login, the IdP generates a SAML Assertion Document and redirects the browser to the SP ACS Endpoint.
4. The SP validates the assertion, sets an encrypted authorization cookie in the browser and redirects the browser to the originally requested URL.
If the Cookie is valid:
1. The content is either served from the cache or retrieved from the origin.

Add an Application in IAM Identity Center

We assume that you already have IAM Identity Center setup, perhaps already with some groups and users.

Open the AWS Console
Open IAM Identity Center
On the left menu, expand "Application assignments" and click on "Applications"
Click on the "Add application" button on the right
Choose "Add custom SAML 2.0 application" and click on the "Next" button
Enter a user friendly Display Name
enter a user friendly Description
Download the IAM Identity center SAML metadata file, you will need it to configure the solution
Leave Application start URL and Relay state empty
Enter dummy values for Application ACS URL and Application SAML audience, we will come back to configure this later
Click on the "Submit" button
Open your application settings
On the "Action" drop-down top right, choose "Edit Attribute Mappings"
Enter the name of your project (this value isn't used, but needs to be filled)
Set the Format to transient
Click on the button "Save changes"

The code: Cloudfront and S3 hosted website

The solution is deployed using serverless.com

It creates a Cloudfront distribution
It creates an S3 Bucket
It creates an SSL Certificate in ACM
It creates an Entry in Route53
It creates 3 Lambda@Edge functions:
- Protect:
- Attached as viewer-request to the default Cloudfront Behavior
- Invoked on each request to validate the cookie and redirect the browser to the IdP if needed
- acs:
- Attached to as viewer-request to the /saml/acs path
- Invoked with POST by the browser with the SAML Assertion Document
- metadata:
- Attached to as viewer-request to the /saml/metadata.xml path
- Allows to retrieve the SP metadata document to configure the IdP
It uses one NPM module:
- samlify

Prerequisites

AWS IAM Roles

Your account needs to have 2 IAM Roles set-up to allow deploying StackSets for the bucket that is in another region:

AWSCloudFormationStackSetAdministrationRole
AWSCloudFormationStackSetExecutionRole

Create them if they don't already exist

AWSTemplateFormatVersion: 2010-09-09
Description: Configure the AWSCloudFormationStackSet Roles to enable use of AWS CloudFormation StackSets.

Resources:
  AdministrationRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AWSCloudFormationStackSetAdministrationRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service: cloudformation.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: AssumeRole-AWSCloudFormationStackSetExecutionRole
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - sts:AssumeRole
                Resource:
                  - "arn:*:iam::*:role/AWSCloudFormationStackSetExecutionRole"
  ExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AWSCloudFormationStackSetExecutionRole
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              AWS:
                - !Sub arn:aws:iam::${AWS::AccountId}:root
            Action:
              - sts:AssumeRole
      Path: /
      ManagedPolicyArns:
        - !Sub arn:${AWS::Partition}:iam::aws:policy/AdministratorAccess

Domain

Your domain needs to be hosted in Route53
You need to retrieve the HostedZoneId

Development tools

Some tools are needed, install them if you don't already have them:

npm i -g serverless
nvm

Get the source

git clone https://github.com/DanielMuller/Cloudfront-Auth-IAM-Identity-Center
cd Cloudfront-Auth-IAM-Identity-Center
npm i

Edit serverless.yml, change the following settings:

service
custom.domainName: This is the domain for your Cloudfront distribution
custom.hostedZoneId: The Route53 HostedZoneId for your domain
custom.bucketRegion: The region you want your S3 Bucket to be created in
custom.sourceToken: Any string, protects direct access to your bucket
custom.origin.DomainName: Remove .${self:custom.bucketRegion} if your bucket is in us-east-1
provider.deploymentBucket.name: Set your deployment bucket name
The region needs to be set to us-east-1, Lambda@Edge functions can only be deployed to this region

Create src/secrets/main.ts:

export const secrets = {
  initVector: '1234567890123456', // change me to any 16 Bytes string
  privateKey: '12345678901234567890123456789012', // change me to any 32 Bytes string
  audience: 'audience string', // change me to my application name
  idpMetadata: `XML content of IdP metadata.xml downloaded earlier`
};

Replace WantAuthnRequestsSigned="true" with WantAuthnRequestsSigned="false" in the IdP metadata if it is present.

Deploy it

sls deploy

Configure your Application in IAM Identity Center

Download your SP metadata.xml by visiting https://yourdomain.example.com/saml/metadata.xml
Open the previously created Custom SAML Application in IAM Identity Center
On the "Action" drop-down top right, choose "Edit Configuration"
Upload your metadata.xml in the "Application metadata" section
Add users or groups to your application

Add some files to your bucket

echo 'Index' > index.html
echo "Page1" > page1.html
echo "Page2" > page2.html
echo "404" > 404.html
echo "403" > 403.html

# Replace ${DomainName} with your bucket name (same as domain name)
aws s3 cp --cache-control max-age=31536000 index.html s3://${domainName}/index.html
aws s3 cp --cache-control max-age=31536000 page1.html s3://${domainName}/page.html
aws s3 cp --cache-control max-age=31536000 page2.html s3://${domainName}/page2/index.html
aws s3 cp --cache-control max-age=31536000 404.html s3://${domainName}/404.html
aws s3 cp --cache-control max-age=31536000 403.html s3://${domainName}/403.html

rm *.html

What did we achieve?

Access to your distribution requires an authentication
SP initiated and IdP initiated login to your site
Direct access to your bucket is blocked
All visitors share the same cached content

Test it out

Open http://${domainName}.s3-website.${aws_region}.amazonaws.com/page.html: You receive an Access denied
Open https://${domainName}/page.html: You will be redirected to the IAM Identity Center login page and back to your site where the page will display
Clear your cookies
Open your IAM Identity Center start page and log-in
Select your application and click on it. You will be redirected to your site and already logged in
Open https://${domainName}/page2. You will see the content of page2/index.html
Open https://${domainName}/anything. You will see your 404 page

Why Lambda@Edge and not Cloudfront Functions?

Cloudfront Functions doesn't have access to the request body. IAM Identity Center will issue a POST request to the ACS endpoint with the payload in the body.
Cloudfront Functions can't include modules and has some limitations on what Javascript functions can be used

Conclusion

Serving your website using Cloudfront is the perfect way to ensure a low latency to all your visitors. You want to use the same technology in development than in production, but you want the next version of your application to be a surprise to your members.

You want to allow only certain members of your organization to have access to the next version.

To achieve this, we added an additional layer to Cloudfront by protecting all access with a login-wall and thus keeping unwanted eyes away from our future release. By adding this to Cloudfront (which is our Network Stack), we didn't need to alter our application's code in any way, keeping it identical for development and production.

By leveraging the Identity Provider already used to grant access to AWS resources, we don't have to manage a separate IdP and still have the fine grained capability of picking which members of the Organization can access our application. No need to send passwords via email and rotate them when we want to revoke an access, like it would have been needed with HTTP's Basic Authentication.

We can as easily revoke access by simply removing the user from the relevant group.

My First Lambda Container

Daniel Muller — Tue, 08 Dec 2020 00:11:20 +0000

Container support for Lambda was announced at Re:Invent 2020. At first, I didn't think much of it. My projects are generally small enough to work fine with a little npm i. But I have a little side project that could benefit from it: running Spamassassin inside Lambda to tackle SES's spam tagging failure.

Despite not being very familiar with Docker, I pulled my sleeves back and started on the project. This put me through the typical 4 stages of an AWS Re:Invent announcement:

😍 Excitement
😞 Disappointment
😡 Anger
😑 Resignation

My Journey

Stage 1: Excitement

Being able to use Lambda in my SES flow without calling a third service, what a relief. Building a container for Lambda shouldn't be that hard...

Lucky for me, Danilo Poccia did a great blog post on how to build your own image from scratch.

I used the spamassassin image provided by instantlinux to get me started. I chose this image, because it runs on Debian, configures Razor, Pyzor and DCC correctly.

I just needed a NodeJS Spamc implementation in my function, pass the body of the E-Mail to Lambda et voilà.

I fired up the container locally, threw some messages at it and there it was, all the messages that SES let through where putting Spamassassin's score into the dark red. I finally had a solution that was cheap (aka pay only when needed) and without any change in my current flow, except adding a lambda.invoke.

At that point, I had big plans for a v2 with EFS backed storage to share Bayes and manual learning of falsely classified messages. One can dream big...

Stage 2: Disappointment

I pushed my image into ECR (Thank You ecr-credential-helper, your tool made my life easier), quickly configured a Lambda manually through the console and invoked it with a test content.

That's when I fell from excitement into disappointment. Everything that worked so perfectly locally, just failed in a big ball of fire. My container in Lambda had the same constraints than any function in Lambda: it runs on a read-only system. I was now unable to run sa-update, unable to bind spamd to 127.0.0.1:783. No pid files could be written, no log files per services, ...

Stage 3: Anger

I naively thought that I could do what I want with my container. No Lambda runtime would go into my way. I was first very angry at AWS for promising me something that didn't meet my wishes/hopes (I generally always somehow land into this state with a new service).

Then I got angry to myself for not having foreseen the obvious, it is logical that my container runs in read-only. It's still Lambda.

I started to try to move everything to /tmp, which kind of worked after several trial-and-error. I don't know how to emulate this in my local Docker, so I had to test online each little tweak. I was still hitting a problem, running spamd on a file socket instead of an io socket, I somehow never got it to work. I wanted to keep spamd, since it gave me a nice output with just the score and the rules.

Stage 4: Resignation

After a few hours, I caved in and let go of my dreams of a fleet of spamd with shared config and rules.

I decided to just run spamassassin and parse the headers on the modified message. sa-update runs during image build and logs are sent to stderr. I disabled DCC, hoping that the standard RBL rules in addition to Pyzor and Razor should be a good enough start. Can't be worse than what SES is doing.

Does it work?

I have this in production since less than 2 days at time of writing, not enough data to make a definitive statement, but so far it behaves very nicely. All the spams are discarded, and no false positive so far. 🤞

The function runs between 10 and 12 seconds and uses between 180 and 220 MB (~0.01¢).

You can find it on Github or on Public ECR.

What is still missing

Despite the read-only limitation, which will probably never be lifted, you currently can't use an image from another account's ECR. You also can't use an image published to public ECR. I guess this are just limitations on a brand new service, and should be lifted in the near future. For now, I just need to push my image to the 2 accounts where they are needed. Some CodePipeline would come in handy.

What's next ?

I could use an EventBridge Scheduler to rebuild the image weekly to have the rules up-to-date. Which anyway is a better solution than pulling them at every start.

Else, I had thoughts about running this on Fargate before, but I didn't insist enough on trying to get down to 0 when there is nothing to process.
AWS batch could be another contender, now that it runs on Fargate.

In Conclusion

Lambda Container won't become my major way of deploying Lambda. I just find the hassle of building, deploying, testing your container more troublesome than an sls invoke local and sls deploy.

But I can see why this will appeal to some peoples, specially when you need tools compiled that you can't compile on your development machine (sharp comes to mind), or if you can't just npm, pip or go get in your dependencies.

AWS SES Receive: It's Worse Than You Thought

Daniel Muller — Sun, 06 Dec 2020 23:24:05 +0000

AWS generally has great products, I use them almost exclusively, both professionally and privately, I am a what you could call a fanboy.

But now and then, they offer a service that is just utterly garbage, which becomes useless unless you write a lot around it. B2C services providfed by AWS often fall into this category.

One of this services is SES Receiving.

I migrated from a self-hosted solution to AWS-SES-Receiving a few weeks back. My main area of concern is the lack of SPAM filtering.

I gathered enough data to realize that this service is very faulty, under performing and creates more problems than it solves.
It other words: it's useless or one could even say: total garbage.

The bad stuff

This are limitation I was aware of before starting my migration and accepted them. But it would make the service so much more appealing if they where addressed.

No IMAP client

AWS has the WorkMail service, which comes with a Webapp and IMAP server and costs 4.00 USD/month/user. For 6.00 USD/month/user, you get a Google Workspace account account, which includes so much more than just EMail (Drive, Meet, ...)

This isn't a deal breaker for me, since I only wanted to provide forwards to personal emails.

No ability to forward messages

You have to send as a user from your domain, so no ability to forward an incoming message keeping the original external sender.

This is the main feature of my needs. I made peace with it, and rewrite the From field and providing a Reply-To.

No in-built alias management

Everybody needs aliases. Be it only info. No way to do this in SES, unless you fiddle with the rules settings.

Not a deal breaker, since I was to handle externally all forwards.

The totally broken stuff

This is the part that made me cringe the most.

Spam filtering

SES Receiving is absolutely worthless when it comes to SPAM filtering. Their functionality is totally useless, an important proportions of SPAM messages are still going through.

What is happening

On average, 30 to 50 messages are received a day. The total sample rate is of 840.

Out of the 840 messages, SES identified:

Spam: 198
Invalid SPF: 2
Ham: 640

On manual analysis (just checking the Sender and Subject), from this 640, I identified:

False Negative: 559
Ham: 81

From the false negative, 33 where dropped with a simple MX check on the sender's domain. Which leaves us with a delivery of 526 Spam messages to my end users.

87.3% of the messages claimed to be clean by SES are actually SPAM. This is an astonishing bad and shameful result.

How to improve this?

There are a few obvious "tricks", which I started to implement:

Check the sender's MX. If it doesn't exist, drop the message. This reduced the SPAM in only a minimalistic way.
Use Postmarks's Spamassassin API: It removed some SPAM, but not enough. Their rules seems top be a bit outdated.
Run your own Spamassassin: which makes us going back to square one and hosting a solution on a server. Yes there is Fargate, or even Lambda Container, but you still need to build and manage it.

What AWS should do

AWS want's to proclaim themselves as a leader in ML. Then show us how! Give us a decent and bullet proof SPAM filtering solution.

We should be able to see some kind of confidence level and decide ourselves what is spam or not. We should even be able to cut the STMP communication early if the level is too high, and never have to process an obvious SPAM messages.

And also, allow us to call an API to declare what is SPAM and what is HAM, like any Gmail Account allows us to do.

In Conclusion

Almost 90% of the messages (yes that's 9 out of 10) claiming to be clean are actually SPAM. This doesn't check anybody's level of expectation.

Until AWS put's a lot of effort into SES, don't use them for your E-Mail solution. Just give your money to Google.

Using AWS S3 as a database

Daniel Muller — Thu, 05 Nov 2020 00:00:15 +0000

During re:Invent 2017, AWS announced a new feature for S3: s3-select, which went GA in April 2018.

In a Nutshell

S3 select allows to retrieve partial content from a single key in S3 using SQL. You can think of it as a single table-database.

There are some constraints:

Data needs to be in a single file and can't be grouped under a prefix
The content of the file must be in JSON, CSV or Apache Parquet
Supported compressions are GZIP or BZIP2
You can't use joins between tables
You can't update a single entry, you need to replace the whole file
- But this comes with a benefit: updating the data becomes very easy. All you need is an S3 PutObject access (Console, CLI, SDK, SFTP, ...)

This is specially useful in bigdata. Instead of loading a large file, you retrieve only the useful content. Therefore reducing network transfer, storage and memory on the processing side, which translates into cost reduction.

S3-Select stills needs to scan the whole file (you pay for that), but you gain on the processing side.

Which format to choose?

Each format has it's pros and cons. The format to use mainly depends on your production capabilities and query needs.

Being text formats, CSV and JSON a very easy to produce (code, spreadsheets, ...). Parquet needs some additional skills and tools.

Schema alterations are straight forward in JSON, since every entry carries it's own schema. For CSV and Parquet, you need to alter each entry with the new field. Using a spreadsheet software helps for CSV.

With all the attributes repeated, JSON is the heaviest format. Half of the file is schema definition. With CSV and Parquet you define your attributes only once. Obviously, this becomes less of a factor once compressed.

On the query side, Parquet is the clear winner. Being a columnar format, the amount of data scanned is greatly reduced, since only the needed columns are used. With CSV and JSON the whole file needs to be scanned.

	JSON	CSV	Parquet
Data Creation	Easy	Easy	Difficult
Data Modification	Easy	Average	Difficult
Storage Size	High	Low	Low
Query speed	Slow	Slow	Fast

Which compression to choose?

You should compress your files, there is no reason to keep them vanilla. Smaller files results in less storage used and faster transfer times from your data source to S3.

BZIP2 is a bit slower, but will generate smaller files than GZIP. You reduce even more your upload times and storage costs, but with a small impact on query times.

A simple example

Dataset used: atp-world-tour-tennis-data/players
Size: 1.5MB (372KB gzipped)
Entries: 10912
Fields: 20

player_id	player_slug	first_name	last_name	player_url	flag_code	residence	birthplace	birthdate	birth_year	birth_month	birth_day	weight_lbs	weight_kg	height_ft	height_inches	height_cm
a002	ricardo-acuna	Ricardo	Acuna	http://www.atpworldtour.com/en/players/ricardo-acuna/a002/overview	CHI	Jupiter, FL, USA	Santiago, Chile	1958.01.13	1958	01	13	150	68	"5'9"""	69	175
a001	sadiq-abdullahi	Sadiq	Abdullahi	http://www.atpworldtour.com/en/players/sadiq-abdullahi/a001/overview	NGR			1960.02.02	1960	02	02	0	0	"0'0"""	0	0
a005	nelson-aerts	Nelson	Aerts	http://www.atpworldtour.com/en/players/nelson-aerts/a005/overview	BRA		Cachoeira Do Sul, Brazil	1963.04.25	1963	04	25	165	75	"6'2"""	74	188
a004	egan-adams	Egan	Adams	http://www.atpworldtour.com/en/players/egan-adams/a004/overview	USA	Palmetto, FL, USA	Miami Beach, FL, USA	1959.06.15	1959	06	15	160	73	"5'10"""	70	178

...

Query the dataset

The API allowing to query a file's content is SelectObjectContent. Also available in most of the SDK's.

For the CLI, you find it under aws s3api select-object-content.

Let's do it with Javascript:

const AWS = require('aws-sdk')
var credentials = new AWS.SharedIniFileCredentials({profile: 'default'})
AWS.config.credentials = credentials
const s3 = new AWS.S3({region:'us-east-1'})

const query = 'SELECT s.last_name,s.birth_year from s3object s where s.flag_code=\'SUI\' and s.birth_year!=\'\''

const params = {
  Bucket: 'myDataBucket',
  Key: 'datasets/atp-players.csv.gz',
  InputSerialization: {
    CSV: {
      FileHeaderInfo: 'USE'
    },
    CompressionType: 'GZIP',
  },
  OutputSerialization: {
    CSV:{}
  },
  ExpressionType: 'SQL',
  Expression: query
}


s3.selectObjectContent(params).promise().then(res => {
  let records = []
  let stats = {}
  res.Payload.on('data', function (event) {
    if (event.Stats) {
      stats = event.Stats
    }
    if (event.Records) {
      if (event.Records.Payload) {
        let data = event.Records.Payload.toString()
        records = [
          ...records,
          ...data.split('\n').filter(l => l).map(e => {
            return {
              last_name: e.split(',')[0],
              birth_year: e.split(',')[1]
            }
          })
        ]
      }
    }
  })
  res.Payload.on('error', function (err) { console.log('err', err) })
  res.Payload.on('end', function () {
    console.log(records)
    console.log(stats)
  })

})

records:

[
  { "last_name": "Allegro", "birth_year": "1978" },
  { "last_name": "Bastl", "birth_year": "1975" },
  { "last_name": "Bohli", "birth_year": "1983" },
  { "last_name": "Blatter", "birth_year": "1949" },
  ...
]

stats:

{
  "Details": {
    "BytesScanned": 379951,
    "BytesProcessed": 1557430,
    "BytesReturned": 1034
  }
}

S3-Select had to scan the whole compressed file and needed to process the uncompressed content. But needed to return only 1KB (~0.3% of the total file).

In conclusion

Considering S3 as a Database is surely far fetched. But it works very well for single large datasets on which you need to retrieve a small chunk. Consider it as your read-only slave database.

Keep in mind, that the read speed is far below what you can achieve with a real databases on SSD, so don't use this for anything time sensitive.

The ability to update content with more traditional non-database tools, allows a wider range of peoples to curate and maintain the data.

A real life example

When S3-Select came out, I decided to use the above concept to create a Serverless GeoIP API using Maxmind's GeoIP2 databases. I wanted to replace the locally installed databases on each server by an HTTP service. This helps on the administrative side by managing updates in only one place, remove boot up scripts while autoscaling, and also it allows other processes and services to use this data (ETL, logging, lambda, ...).

You can read more about it in my blog or install it from my Github repo (works with GeoIP2 and GeoLite2 databases).

I made some tweaks from the above example:

The data files are sharded by primary key to reduce the scan size
Cloudfront is used as a database-cache layer to avoid having to query twice for the same IP.

Why not DynamoDB?

When S3-Select came out, DynamoDB didn't have an "On-Demand" pricing. Having to provision (and pay for) a fixed throughput was too expensive.

Today, with "On-Demand" pricing, I would definitively use DynamoDB for this tool. Storage pricing is similar to S3, query pricing also. Maintaining and querying the data in DynamoDB is also easier.