Context
In large companies with half a dozen or more development teams, it's common to have each team to control its own infrastructure and deploy services independently.
When I joined a startup, I initially maintained the same approach: one endpoint per service. However, I recently reorganized the code, consolidating everything into a single Git repository:
- An HTTP API (using AWS API Gateway).
- A Websocket API (using AWS API Gateway).
- A static website (hosted on S3 and served via CloudFront).
- A CDN for persistent data (hosted on S3 and served via CloudFront).
- All configured and deployed using AWS CDK and GitHub Actions.
Why consolidate all CloudFront distributions?
Last September, I came across an interesting suggestion in Yan Cui's post on X (formerly Twitter):
- If you host your APIs or other resources on different domains (even subdomain), browsers make additional calls to those endpoints to meet CORS requirements.
- Since API Gateway charges per request and data transfer, this effectively means paying for two requests instead of one!
I appreciate the simplicity of deploying a web application where all requests to backend services use relative paths (/api/v1/user/me or /cdn/images/REWR7.avif) instead of fully qualified ones (https://api.example.com/v1/user/me or https://cdn.example.com/images.REWR7.avif). This makes deploying the website virtually effortless, regardless of the environment.
While this situation can be mitigated, as I'll explain later, I decided to expose the APIs and S3 services through the same CloudFront distribution as the static website.
My goals are:
- Reduce costs by minimizing requests to the services.
- Reduce latency by minimizing the number of requests that need to be answered.
- Conserve network resources by maintaining only one open HTTP/2 or HTTP/3 connection.
Critical enablers
Successful consolidation of multiple services within a single CloudFront distribution requires that there be no overlapping paths used to access those services.
In my case:
- The paths for the APIs did not conflict with any resource path deployed with the website. The path pattern is like:
/v{version}/{scope}/{resource} - The resources in S3 already had unique prefixes (aka folder names in S3-speak) such as
/data,/assets, etc. - The website exposes files at the root level (like
/index.html), in standard folders (like/.well-known), and a/resourcesfolder.
With the help of the AWS CDK
I much prefer writing code with the AWS CDK over scripting in Jenkins or writing YAML for Terraform. Infrastructure as Code (IaC) with the CDK is fantastic!
With a couple of files, I defined the setup and behaviors of:
- A few tables in AWS DynamoDB for transactional data.
- One S3 bucket for persistent data.
- One Lambda function for each major HTTP API resource handler.
- One Lambda for the WebSocket authorizer and another one for its handler.
- And many other Lambda functions: for the scheduled jobs (EventBridge), for the state machine (AWS Step Functions), for the message queue management (AWS SQS), for transaction analysis (DynamoDB Streams), etc.
- Two static websites (one public, one for administrators) hosted on S3.
The code samples I'm going to share below are written in TypeScript and work with the CDK v2.184. I have only made minor edits to match the domain https://example.com.
The challenges
For a seamless transition, I aimed to maintain the existing CloudFront distributions with robust CORS support.
Consolidating into a single distribution must not compromise access control. I have one API secured by AWS Cognito, using a user pool for email-authenticated users, and another API with a custom authorizer for users authenticated via a third-party OpenID Connect provider.
The existing caching strategy for each service and S3 bucket must be preserved.
The IaC code update
For brevity, I'll only share the setup for the HTTP API, as the other setups are quite similar.
The configuration for the HTTP API and its own CloudFront distribution
// HTTP API
this.httpApi = new HttpApi(this, 'ExampleHttpApi', {
corsPreflight: {
allowCredentials: true,
allowHeaders: props.allowedRequestHeaders,
allowMethods: [CorsHttpMethod.ANY],
allowOrigins: props.allowedOrigins,
exposeHeaders: props.allowedResponseHeaders,
maxAge: Duration.hours(2),
},
description: 'HTTP API',
disableExecuteApiEndpoint: false, // Because no custom domain, just CF
});
The API definition is set to issue the responses to the CORS preflights automatically. Accesses via CloudFront will see the response cached and served automatically during the next two hours.
// To let CloudFront handle the CORS-related headers
const corsHeadersPolicy = new ResponseHeadersPolicy(this, 'CorsHeadersPolicy', {
corsBehavior: {
accessControlAllowCredentials: true,
accessControlAllowHeaders: props.allowedRequestHeaders,
accessControlAllowMethods: ['ALL'],
accessControlAllowOrigins: props.allowedOrigins,
accessControlExposeHeaders: props.allowedResponseHeaders,
accessControlMaxAge: Duration.hours(2),
originOverride: true,
},
});
// CloudFront Distribution
const distribution = new Distribution(this, 'ExampleHttpApiDistribution', {
comment: 'HTTP API front',
defaultBehavior: {
allowedMethods: AllowedMethods.ALLOW_ALL,
cachedMethods: AllowedMethods.ALLOW_GET_HEAD_OPTIONS,
cachePolicy: CachePolicy.CACHING_DISABLED,
compress: false,
origin: new HttpOrigin(`${this.httpApi.apiId}.execute-api.${Stack.of(this).region}.amazonaws.com`),
originRequestPolicy: OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER,
responseHeadersPolicy: corsHeadersPolicy,
viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
},
});
// Force the dependency so `apiId` is available this CF setup starts
distribution.node.addDependency(this.httpApi);
Note the origin request policy ALL_VIEWER_EXCEPT_HOST_HEADER is recommended by AWS for API Gateway and Lambda function origins.
The configuration for the static website and its own CloudFront distribution
// S3 bucketCloudFront redirects requests to known endpoints
const bucket = new Bucket(this, 'ExampleWebsiteBucket', {
accessControl: BucketAccessControl.PRIVATE,
blockPublicAccess: BlockPublicAccess.BLOCK_ALL,
// cors: [], // No CORS as it interferes with Origin Access Control (OAC)
enforceSSL: true,
removalPolicy: RemovalPolicy.DESTROY,
});
As you can see, the bucket is being given an automatic name by CloudFormation. The name is not important here because no script will ever use the AWS SDK to read or update the bucket content. The bucket is also set with the removal policy set to DESTROY because its content can be recreated anytime.
Note that I take a different strategy for the bucket that serves as my CDN:
- The removal policy is set to
RETAINbecause I don't want to lose the data accumulated over time. - The bucket is named so the scripts can rely on a name known in advance.
- The bucket is part of a backup (also set with the CDK).
// CloudFront distribution
const distribution = new Distribution(this, 'ExampleWebsiteDistribution', {
certificate,
comment: 'Public website',
defaultBehavior: {
allowedMethods: AllowedMethods.ALLOW_GET_HEAD_OPTIONS,
cachePolicy: CachePolicy.CACHING_OPTIMIZED,
compress: true,
origin: S3BucketOrigin.withOriginAccessControl(bucket, {
originAccessControl: new S3OriginAccessControl(bucket, 'WebsiteOriginAccessControl'),
originAccessLevels: [AccessLevel.READ], // No `LIST` access for the website content
},
originRequestPolicy: OriginRequestPolicy.CORS_S3_ORIGIN,
responseHeadersPolicy: corsHeadersPolicy, // Same as above
viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
},
defaultRootObject: 'index.html',
domainNames: ['app.example.com'],
errorResponses: [
{
httpStatus: 403, // When a S3 returns `Access Denied`
responseHttpStatus: 200,
responsePagePath: '/index.html', // SPA access point
ttl: Duration.minutes(30),
},
{
httpStatus: 404, // If S3 returns a regular `Not Found`
responseHttpStatus: 200,
responsePagePath: '/index.html', // SPA access point
ttl: Duration.minutes(30),
},
],
httpVersion: HttpVersion.HTTP2_AND_3,
minimumProtocolVersion: SecurityPolicyProtocol.TLS_V1_2_2021,
});
The distribution has two particularities: it changes the S3 HTTP status code from 403 or 404 to 200 when S3 cannot serve a request, and it returns the content of the root object.
The extension of the CloudFront distribution for the static website
distribution.addBehavior(
'/v2.23/*',
new HttpOrigin(`${props.httpApi.apiId}.execute-api.${Stack.of(this).region}.amazonaws.com`, {
keepaliveTimeout: Duration.minutes(1),
customHeaders: {
'X-Use-444-Not-Found': 'true', // For the API to return 444 code in place of 404 code
},
}),
{
allowedMethods: AllowedMethods.ALLOW_ALL,
cachePolicy: CachePolicy.CACHING_DISABLED,
compress: false,
originRequestPolicy: OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER,
viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
},
);
The configuration to serve the requests for the HTTP API from the same distribution as the website (that means via https://app.example.com) has one specific setting:
- A custom header set by CloudFront in all requests at the destination to the HTTP API—more information in the section below.
The behavior definition also clearly states that the HTTP API responses should not be cached. The payload produced by the HTTP API handler can contain headers like Cache-Control. They only have an effect on the end-user client (a browser or a smart native application).
Note also the distribution is set to not compress the response to reduce the latency. Usually, the HTTP API responses are small enough that compressing them even with brotli is not worthwhile.
Lessons learned
CORS protocol
I initially believed that cross-origin requests always involved a preflight request. The goal of this request is to inform the browsers if the remote resource allows any data use for that origin. The response of the preflight requests is often set with a Cache-Control header that allows browsers to trust the service response for a little while. This caching strategy saves bandwidth and improve the over-the-wire communication performance in the browser.
It turns out that browsers can skip preflight requests for simple requests. For the browser to process the response, this one must contain the header Access-Control-Allow-Origin with either * or the request issuer's origin (like https://app.example.com).
Note that for most services, using '*' or echoing the request's origin is discouraged due to the risk of XSS attacks and potential data leaks to uncontrolled origins. It's better to answer with one accepted value only.
404 — Not Found
In a standalone REST API, Lambda functions invoked by the HTTP API often return a 404 Not Found status when no resource matches the request.
If this behavior is left intact, the client using the endpoint common with the website (i.e., https://app.example.com) will get responses with the HTTP status 200 OK and the content of the /index.html file as the payload.
This behavior is expected in Single-Page Applications (SPAs):
- When the user first visits the page with the base URL https://app.example.com, the browser receives the content of the
/index.htmlpage. - As the user visits more content in the application, the URL changes to keep track of the context. It can take a value like https://app.example.com/course/DSWL/chapter/3.
- At any moment, the user can refresh the browser page, or it can bookmark it to come back later at the same place.
- Since no HTML resource exists at that path, the S3 API will return a
403 Access Deniederror to CloudFront!" - To avoid a disruption, we want CloudFront to return the content of the
'/index.htmlpage, trusting the application logic to make sense of the URL and to restore the content for the end user.
Because the HTTP API is likely accessed by a JavaScript fetch() call in a browser or HTTP requests from a native client, we need a mechanism to convey the 404 Custom Error information.
The solution I adopted is to let the Lambda functions behind the HTTP API know that we want a different error code than 404 Not Found when a resource cannot be found. In my case, I opted for 444 Custom Error, hence the header X-Use-444-Not-Found set to true in the CloudFront distribution definition.
Within the HTTP API Lambda handlers, during request payload processing, I invoke the following helper function to set a static variable. This variable is then used by the NotFoundException to generate the appropriate error code.
function _adjustBaseExceptionNotFoundErrorCode(headers: { [key: string]: string }): void {
const controlHeader: string = 'X-Use-444-Not-Found'.toLowerCase();
const detectedControlHeaders: string[] = Object.keys(headers).filter((key: string): boolean => {
return (key.toLowerCase() === controlHeader && this.headers[key]) === 'true';
});
IBaseException.use444NotFound = 0 < detectedControlHeaders.length;
}
The client implementation was modified to include the 444 Custom Error within the error handling logic, in conjunction with the standard 404 Not Found.
Note the X-Use-444-Not-Found custom header preserves standard 404 Not Found behavior in Lambda functions accessed directly or via standalone CloudFront distributions within this consolidated setup.
Optimizing the HTTP/2 HTTP header management
On the HTTP/2 protocol (and its successor, HTTP/3), the full set of headers are passed over the wire with the first request. With subsequent requests, updated header values are transmitted to the server as compressed differences.
Typically, when exchanging data with a CDN, authorization tokens are not included in header values.
With this consolidated CloudFront distribution, if the web application alternates between CDN and API requests, the browser will frequently update the header payload, alternately removing and restoring the Authorization header.
To optimize processing, it's preferable to consistently include the Authorization header with the authentication token.
This is similar to always sending Accept: application/json, plain/text, even when the application primarily expects JSON, with plain text only returned after a successful POST request with a 201 Created status.
How to mitigate the issue without much refactoring
Making services like an API or an S3 bucket accessible via a different domain doesn't have to be costly. Here are several options for cost optimization:
- Utilize simple requests whenever possible.
- Leverage CloudFront in front of API Gateway or S3 buckets:
- The first 1TB of monthly data transfer out is free.
- Data transfer out from CloudFront is generally less expensive than from S3 or API Gateway.
- CloudFront traffic largely stays within the AWS private network, whereas direct traffic to S3 or API Gateway traverses the public internet.
- CloudFront edge location caching enhances performance.
- Configure appropriate
MaxAgeSecondsto cache preflight responses. Note that Chromium-based browsers limit this to a maximum of 2 hours. - Employ WebSocket connections instead of REST API calls for frequent communication.
Conclusion
In this post, we've demonstrated how to consolidate multiple services under a single CloudFront distribution using the AWS CDK. This method streamlines infrastructure management, lowers costs, and boosts performance. Utilizing custom headers and error handling, we can efficiently serve both static content and API endpoints through a unified distribution. While security and monitoring remain essential considerations, the advantages of consolidation make it a compelling strategy for numerous applications.
What are your experiences with consolidating CloudFront distributions? Share your thoughts in the comments below!
Top comments (0)