Cloudformation templates for Cloudfront automatic cache invalidation using Lambda within CodePipeline

(original post: https://marcoaguzzi.it/2024/01/03/lambda-invalidation-cloudformation/)

In this post I’m going to show how I triggered an automatic cache invalidation for the Cloudfront distribution that is serving this website. As in the previous posts, all the resources will be provisioned via CloudFormation.

At the end of the post the CLI commands to create and / or update the resources will be shown.

The manual procedure

Once that the markdown file for a post is written and a local compilation / rendering has been made, the markdown source can be pushed on the git repo. That triggers the AWS Codepipeline that will download the source, render the markdown into html, and push the result to the S3 bucket served by Cloudfront.

Since Cloudfront is serving the S3 bucket, caching is in place. Newly pushed content won’t be visible until the cache expires, which is not feasible. So, after a successful compilation and pushing to S3, I manually get to Cloudfront distribution invalidations and fire a new invalidation. This way I’m sure that subsequent requests to the website will get the newly updated content.

In the images below the steps for manual invalidation are shown:

Go to CloudFront / Distributions, and search for “Invalidations” tab

Then selecting the last successful invalidation (shown below on the very left) and “copy to new” (upper right)

And then confirming the copy of the invalidation with the last path (the path /* is fine since AWS charges per invalidation, regardless of how much deep it is)

The invalidation takes a few minutes to be completed, and then the website is good to go. This is a mundane and forgetful-prone task, so I’m better automating it.

Automation setup

There is not an “invalidate cache” action that can be directly call from CodePipeline. A Lambda that actually creates the invalidation is needed and must be called as an action in the CodePipeline structure.

Let’s see in details the two resources:

The Lambda function

The Lambda function will leverage boto3 Python libraries to create the invalidation and notify the pipeline about the outcome (credits to the website in the reference section).

Let’s see some highlights. At the end of this section the link to the gist with the full source is provided.

• Read cloudformation ID from environment

• Caller notifications

• Invalidation specifications

Click to view the Gist with the Lambda Python code

Lambda Cloudformation stack and how to reference it

The Lambda cloudformation stack is similar to the one presented for the 301 redirects and URL rewriting in edge locations (here’s the post); there are a few differences, though (at the end of the paragraph there is the gist with the full code):

• In the lambda’s resources, an environment variable is declared and its value is read from the cloudformation stack that contains the cloudfront distribution (referenced in the python code as CLOUDFRONT_DISTRIBUTION_ID). To be able to read that from here, the cloudformation stack that contains the cloudfront distribution has to list the variable as an output and flag it to be exported:

Cloudfront distribution environment variable reference

Parameters:
CloudformationExportVar:
Type: String
...
Environment:
Variables:
CLOUDFRONT\_DISTRIBUTION\_ID:
Fn::ImportValue:
!Ref CloudformationExportVar

And here’s the export in the stack hosting the cloudfront distribution

Exported value from cloudfront distribution resource

"Outputs": {
"CloudFrontDistributionId": {
"Description": "ID of the CloudFront distribution",
"Value": { "Fn::GetAtt": ["CloudFrontDistribution", "Id"] },
"Export": {
"Name": { "Fn::Join": ["-", [{ "Fn::Sub": "${AWS::StackName}-CloudFrontDistributionId" }, { "Ref": "Stage" }]]}
 }
 }
 }

• There is no need for the edge permission (only lambda.amazonaws.com is needed)

services for the AssumeRole action

AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action: "sts:AssumeRole"

• Other than the BasicExecutionRole, three other permissions must be granted for the creation of the invalidation and the notification back to the CodePipeline
  • cloudfront:CreateInvalidation
  • codepipeline:PutJobFailureResult
  • codepipeline:PutJobSuccessResult

Additional permissions for basic lambda role

Policies:
- PolicyName: InvalidateCloudfrontDistributionPolicy
PolicyDocument: 
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- cloudfront:CreateInvalidation
- codepipeline:PutJobFailureResult
- codepipeline:PutJobSuccessResult
Resource: "\*"

That should put the lambda in place for the purpose.

Here below you can view the full gists of the cloudformation stack of

• the lambda#Click to view the Gist
• the cloudfront distribution (which is the parent stack as shown in the older articles)#Click to view the Gist

CodePipeline stage

The action can be added in CodePipeline as a new stage. From there the lambda can be referenced. Here’s how the new stage will look like after cloudformation template has been deployed (You can see that the action is referring to the lambda and still no runs shown):

Codepipeline Cloudformation stack

Let’s start from the addition of the new stage in CloudFormation. We can see

• the parameter that will reference the exported value from the lambda stack. The parameter value contains the name of the exported variable, and will reference the lambda function name
• the action
• the updated permissions in order to allow the calling of the lambda function from the pipeline:

New stage in Codepipeline

"Parameters": {
...
"InvalidationLambdaExported": {
"Description": "Lambda function performing the cache invalidation",
"Type": "String"
 }
}
... stages ...
{
"Actions": [
 {
"ActionTypeId": {
"Category": "Invoke",
"Owner": "AWS",
"Provider": "Lambda",
"Version": "1"
 },
"Configuration": {
"FunctionName": {"Fn::ImportValue":{"Ref":"InvalidationLambdaExported"}}
 },
"Name": "InvalidateCloudFrontCacheAction"
 }
 ],
"Name": "InvalidateCloudFrontCacheStage"
}
...
"Policies": [
 {
"PolicyDocument": {
"Statement": [
 {
"Action": [
...
"lambda:invokeFunction"
 ],
"Effect": "Allow",
"Resource": "\*"
 }
 ],
"Version": "2012-10-17"
 },
"PolicyName": "MyCodePipelineRolePolicy"
 }
]


The full reference to the cloudformation template can be found in the gist below:

#Click to view the Gist cloudformation for the Cloudfront distribution<script src="https://gist.github.com/maguzzi/4899697488d40105dd51ce2c37c1e327.js"></script>

Below is how the new CodePipeline stage should turn out if everything was successful:

 ![Codepipeline stage invalidation succedeed](https://marcoaguzzi.it/img/cache-invalidation-cloudformation/cloudfront_invalidation_5.png)

And that should do for adding a new stage with a new action calling the lambda and invalidating the cache

## Cloudformation CLI commands

Here’s the AWS CLI commands (legit ones!) that have been fired in order to create and / or update the cloudformation stacks (and the lambda, of course):

_CloudFront stack update_

aws cloudformation package --template-file marcoaguzzi.json --s3-bucket cf-templates-e5ht2sji9no7-us-east-1 --output-template-file target\packaged-template.yaml
aws cloudformation deploy --template-file target\packaged-template.yaml --stack-name marcoaguzzi-website-prod --capabilities CAPABILITY_NAMED_IAM --parameter-overrides Stage=prod

It should appear the exported output variable:  
 ![Export from CloudFront](https://marcoaguzzi.it/img/cache-invalidation-cloudformation/cloudfront_output.png)

_New lambda creation, passing the export from above_

cd lambda-cloudfront
compress-Archive .\index.py .\lambda-cloudfront-invalidate-prod-20240101.zip
aws s3 cp lambda-cloudfront-invalidate-prod-20240101.zip s3://lambda-artifacts-bucket-maguzzi/
aws cloudformation create-stack --stack-name lambda-invalidate-cloudfront-prod --template-body file://lambda-invalidate.yaml --capabilities CAPABILITY_NAMED_IAM --parameters ParameterKey=ZipDate,ParameterValue=20240101 ParameterKey=Stage,ParameterValue=prod ParameterKey=CloudformationExportVar,ParameterValue=marcoaguzzi-website-prod-CloudFrontDistributionId-prod

The new stack is created and the lambda name is exported in outputs:  
 ![Export from Lambda](https://marcoaguzzi.it/img/cache-invalidation-cloudformation/lambda_output.png)

_CodePipeline update referencing the lambda exported variable in parameters_

aws cloudformation update-stack --stack-name marcoaguzzi-stack-codepipeline-prod --template-body file://marcoaguzzi-codepipeline.json --parameters file://parameters-codepipeline-prod.json --capabilities
CAPABILITY_IAM

And now the Codepipeline should have the last stage as shown in the pictures above, ready to invalidate the cache after the website deploy to S3 :-)

## References

- [AWS: Creating a CloudFront Invalidation in CodePipeline using Lambda Actions](https://medium.com/fullstackai/aws-creating-a-cloudfront-invalidation-in-codepipeline-using-lambda-actions-49c1fd3a3c31)
- [AWS Cloudformation user guide](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html)