Generating AWS CloudFormation template property values at deploy time

Jerry Mindek — Thu, 23 Sep 2021 16:36:09 +0000

Do you wish you could generate property values for resources in your AWS CloudFormation template at deploy time?

In this post I would like to share with you a step-by-step approach.

My goal is to enable you to do this quickly and to prevent you from a nasty blind-spot that I crashed into!

It all began once upon...

Summer of 2021 while I was creating an AWS Data Pipeline which manages an Apache Spark ETL application and its needed AWS resources.
The Spark application runs in Amazon Elastic Map Reduce (EMR).
So, the Data Pipeline is implemented with an EMR resource and activity with preconditions and post step commands, and an EC2 resource activity to execute some aws CLI tool activities.
I packaged all this up in a nice and tidy CloudFormation template so that I can deploy it from AWS CodePipeline.

Which brings us to the impetus for this post.
I need the Data Pipeline to start daily at 3am.
I don't want to set that value each time I deploy; I want it to be generated.

Initial Solution

At first, I updated the Data Pipeline's default schedule startDate attribute at build time.
In the CFN template, I had

- Name: "Every 1 day"
          Id: "DefaultSchedule"
          Fields:
            - Key: period
              StringValue: "1 days"
            - Key: startDateTime
              StringValue: ||startdate||
            - Key: type
              StringValue: "Schedule"

During execution of the build script in the build process I used sed to convert ||startdate|| to the value calculated a couple lines earlier in the build script.
When the template was packaged with aws cloudformation package it has a valid date time for the Data Pipeline default schedule's startDateTime attribute.

Pros

easy to do
dependable

Cons

A redeploy would use the startDate set by the build, setting the startDate in the past.
Could not deploy by simply uploading the template.

These aren't major issues. However, the solution lacks flexibility and finesse.
I set out to fix that.

The better solution: Setting resource property values with deploy-time generated values

To do this, you will need 3 additional resources in your CFN template:

AWS::CloudFormation::CustomResource resource
AWS::Serverless::Function
AWS::Logs::LogGroup resource

Step-by-step

Create a CustomResource to capture the generated value.
Create a Function with in-line code to generate the generated value
Create a log group to capture log messages from the Function so that you can troubleshoot issues with the it and the related CustomResource.
Use the generated value as a value for some resource property

It's not hard. Here's the YAML for each of these new AWS CFN resources:

PipelineStartDate:
    Type: AWS::CloudFormation::CustomResource
    Properties:
      ServiceToken: !GetAtt PipelineStartDateFunction.Arn

PipelineStartDateFunctionLogs:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub '/aws/lambda/StartDateGenerator'
      RetentionInDays: 3

PipelineStartDateFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: !Sub 'StartDateGenerator'
      Runtime: python3.6
      Timeout: 30
      Role: !GetAtt SimpleLambdaExecutionRole.Arn
      Handler: index.lambda_handler
      Code:
        ZipFile: |
          from datetime import date, datetime, time
          from datetime import timedelta
          import logging
          import cfnresponse
          def lambda_handler(event, context):
            logging.info(f'REQUEST RECEIVED: {event}')
            try:
              tomorrow = date.today() + timedelta(days=1)
              start_date = datetime.combine(tomorrow, time(hour=3))
              logging.info(f'Computed start date: {start_date}')
              responseData = {}
              responseData['StartDate'] = start_date.strftime('%Y-%m-%dT%H:%M:%S')
              logging.info(f'Sending this response data back to Cloudformation: {responseData}')
              cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "CustomResourceStartDate")
            except:
              cfnresponse.send(event, context, cfnresponse.FAILED, {}, "CustomResourceStartDate")
            return

And to use the generated value:

- Name: "Every 1 day"
          Id: "DefaultSchedule"
          Fields:
            - Key: period
              StringValue: "1 days"
            - Key: startDateTime
              StringValue: !GetAtt PipelineStartDate.StartDate
            - Key: type
              StringValue: "Schedule"

Short explanation

What we did here is create a lambda-backed CustomResource.
The value for our default schedule attribute is the the value we get when we !GetAtt the custom resource key "StartDate".
The value is populated by our serverless function (with help from the AWS provided cfnresponse) who's code is defined in-line. This allows us to include the generator inside the CFN template where it is most applicable.

The Blind-Spot

I spent more time than I am willing to admit trying to determine why my initial attempts to do this resulted in deploys that failed after an hour.
It was impossible to debug because I didn't have any log messages from the CustomResource or the PipelineStartDateFunction.
The key to solving the problem was to add a log group to capture messages from the PipelineStartDateFunction.
Then I was able to goto CloudWatch, see the messages in the log group from the function that quickly explained the problem.

Wrap up

I have created several full stack CloudFormation templates, each time I wondered how I could generate a value at deploy time.
Now that I know, I see that it's not that difficult.
My hope is that this post can give you a clear understanding of how to generated property values at deploy time allowing you to make your CloudFormation templates a little less static!

Cheers!

For more detailed information about some of the AWS resources I talk about, I have included several AWS documentation links for you.

Custom Resources
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-cfn-customresource.html

Serverless Functions with InLine code
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html

DEV Community: Jerry Mindek