Goodbye CloudFormation

No, CloudFormation isn’t going away, but I’m absolutely done with it!

I wasn’t planning on writing this article for a while, but after a rough couple weeks at work struggling with CloudFormation (CF) I’m ready to rip it all out of my little weather data project. I’ve decided to use Terraform (TF) as my only IaC tool, including for deploying and managing my Lambda function. Come along for the ride wont you?

Debating the virtues of CloudFormation vs. Terraform is likely akin to the debate between Vim and Emacs. I realize that there are people with strong opinions on both sides of the fence, but I’m finding myself more comfortable on the TF (and Vim) side. Here are my reasons for wanting to move to TF for all AWS resource management (based solely on my personal experience):

1. By default, CF doesn’t care much about resource drift. Sure, you can go digging and find details about how your AWS resources differ from your CF templates, but CF will happily (and quietly) apply updates to your stacks even if you have made changes manually, say in the AWS console. Worse even than not warning about resource drift, CF will not, by default, overwrite manual changes made to resources, thus leaving your CF template out of sync with your actual resource state. If that’s OK with you, well have at it, but I’m NOT cool with that.
2. Sometimes, the changes you make in a CF template will simply not be applied to your existing infrastructure by updating an existing CF stack (even if you haven’t made changes manually outside of your CF template). I experienced multiple situations where I had to tear down a stack and re-create it in order to apply the changes I made to the template. That’s not always going to be a viable option.
3. I’m already using TF to manage my base architecture, so it would be great to use (and learn) a single IaC tool.

One thing I ran into right away is that since AWS only recently started supporting a Java 17 runtime for Lambda functions, I needed to upgrade the version of the AWS provider that my Terraform installation is using. I had assumed that since I hadn’t specified a version when I declared the provider, the latest version would always be used, but this is not the case. You can upgrade your providers to their latest versions by executing the command terraform init -upgrade.

When I was using AWS SAM (and thus CF) to package and deploy the Lambda function, it took care of the packaging details in coordination with Gradle. Now that I’m using TF instead, I had to take some different steps to build the deployable package. I found that the most straightforward way to build a deployable package was to add a buildZip task to my build.gradle file and use that instead of the standard Gradle tasks.

task buildZip(type: Zip) {
 from compileJava
 from processResources
 into('lib') {
   from configurations.runtimeClasspath
 }
}

That was the only change I made outside of the new TF file I created to define the Lambda function and it’s resources. Here’s the final, TF file I used to get the Lambda function deployed and working. I have only added the bare necessities to this file to keep things simple (technically the IAM role and role policy attachment could be created separately, but the Lambda function will need a role ARN in order to be created).

resource "aws_lambda_function" "weather_tracker" {
  // Required arguments
  function_name = var.project
  runtime = "java17"
  handler = "org.weathertracker.Handler::handleRequest"
  filename = data.local_file.lambda_package.filename
  role = aws_iam_role.lambda_exec.arn

  // Optional arguments
  source_code_hash = "${base64sha256(data.local_file.lambda_package.filename)}"
  memory_size = 1024
  timeout = 300
}

data "local_file" "lambda_package" {
  filename = "${path.module}/../build/distributions/${var.project}-${var.lambda-version}.zip"
}

resource "aws_cloudwatch_log_group" "lambda_log_group" {
  // Required arguments
  name = "/aws/lambda/${aws_lambda_function.weather_tracker.function_name}"

  // Optional arguments
  retention_in_days = 7
}

resource "aws_iam_role" "lambda_exec" {
  name = "serverless_lambda"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Sid = ""
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_policy" {
  role = aws_iam_role.lambda_exec.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

A quick look at this file will show that I’ve declared the aws_lambda_function resource and specified the name, runtime, and handler function. I’ve also provided references to the local_file, and aws_iam_role resources. I also added two optional arguments because without defining the memory_size, and timeout values, this definition would create a Lambda function that is allocated 128 MB of memory and has a time out of 3 seconds which will not work for the purposes of the weather data app I’m creating. Next, I’ve declared thelocal_file resource which points at the artifact created by Gradle when I ran ./gradlew buildZip. Then I’ve declared an aws_cloudwatch_log_group with a retention period of 7 days. Technically, the log group isn’t required for TF to create your Lambda function, but if you don’t define your own log group, AWS will create one for you the first time your Lambda function is executed, and the logs will live forever after you’ve destroyed all of your TF managed resources and you may end up with a bill for the log storage. So for all intents and purposes, I’m considering it a requirement.

Finally, I’ve declared the aws_iam_role and aws_iam_role_policy_attachment which will allow the Lambda function to be executed. Note that the order in which these resources is declared is not important. Terraform is smart enough to tease out the dependencies and create the resources in the appropriate order.

I also created a variables.tf file and a DEV.tfvars file which have the respective contents below.

variable "project" {
  type = string
}
variable "lambda-version" {
  type = string
}

project = "weather-tracker-java-rds-lambda"
lambda-version = "0.0.2-SNAPSHOT"

If you’ve read any of the other articles I’ve written about what I’m going to be doing with the weather data collection app I’m creating, you may notice that this simple Lambda configuration isn’t going to work for my needs. At a minimum, I’ll be adding my own Cloudwatch log group, an EventBridge trigger, and additional permissions, but there may also be other changed needed as I progress through the project. So, if you’re following along with the GitHub repo for the project, you’ll probably find that there have been a few changes to this piece of TF configuration.

During the process of implementing the Lambda function definition in Terraform, I found one possible down side of using TF to deploy the lambda. Due to the way TF’s AWS provider is working with AWS APIs, specifying the source_code_hash attribute of the aws_lambda_function resource doesn’t function as expected. I would hope that using this setting would allow TF to only update the Lambda function if the code has changed. Unfortunately, it seems to be updating every time terraform apply is executed. This isn’t a big deal for my purposes, but I’d guess there might be some use cases where that isn’t ideal, or even acceptable. You can read more about this issue on Terraform’s GitHub. I did try removing the source_code_hash argument, but when it’s not there, the only way I found to get Terraform to update the Lambda function is to update the version, and thus package file name. That’s somewhat workable, but doesn’t allow us to update the Lambda function and test new SNAPSHOT versions very easily.

I don’t know about you, but I feel a lot better now, knowing how to deploy a Lambda function using Terraform rather than AWS SAM and CloudFormation.

Was this article helpful? Did you learn something? Was it worth your time, or a waste of time? I’d love to hear from you if you have questions or feedback on this article and/or if I can help you get past any stumbling blocks you may have encountered along the way!

---