Consistently deploying Lambda functions and layers using Terraform

The code that accompanies this blogpost can be found here

Deploying Lambda functions to AWS using Terraform can be quite a struggle, especially when deploying from multiple environments (which only happens in dev and test environments, am I right?).

Some issues you can encounter are:

• Lambda functions redeploying at every terraform apply
• Errors about missing archive files containing the Lambda function files
• Soft locks in the Terraform state file

In this post I’ll be showing you a way to be able to consistently deploy Lambda functions, only when there are changes to the code, when deploying from multiple environments.

History

There have been long time issues when deploying Lambda functions using Terraform. Some external links with examples of these issues and some attempts to tackle them:

• https://stackoverflow.com/q/52662244/19024815

I have an AWS Lambda deployed successfully with Terraform:

resource "aws_lambda_function" "lambda" {
 filename                       = "dist/subscriber-lambda.zip"
 function_name                  = "test_get-code"
 role                           = <my_role>
 handler                        = "main.handler"
 timeout                        = 14
 reserved_concurrent_executions = 50
 memory_size                    = 128
 runtime                        = "python3.6"
 tags                           = <my map of tags>
 source_code_hash               = "${base64sha256(file("../modules/lambda/lambda-code/main.py"))}"
 kms_key_arn                    = <my_kms_arn>
 vpc_config {
   subnet_ids         = <my_list_of_private_subnets>
   security_group_ids = <my_list_of_security_groups>
 }
 environment {
   variables = {
     environment = "dev"
   }
 }
}

Now, when I run terraform plan command it says my lambda resource needs to be updated because the source_code_hash has changed, but I didn't update lambda Python codebase (which is versioned in a folder of the same repo):

 ~ module.app.module.lambda.aws_lambda_function.lambda
 last_modified:                     "2018-10-05T07:10:35.323+0000" => <computed>
 source_code_hash:                  "jd6U44lfe4124vR0VtyGiz45HFzDHCH7+yTBjvr400s=" => "JJIv/AQoPvpGIg01Ze/YRsteErqR0S6JsqKDNShz1w78"

• https://nulldog.com/trigger-terraform-lambda-code-uploads-a-guide

Trigger updates with null_resource:

If you need more control or want to trigger updates based on other resources, use null_resource:

resource "null_resource" "lambda_update" {
  triggers = {
    code_hash = filebase64sha256("my-function.zip")
  }

  provisioner "local-exec" {
    command = "echo 'Code updated, triggering Lambda deployment...'"
  }
}

resource "aws_lambda_function" "example" {
  # ... other configurations
  depends_on = [null_resource.lambda_update]
}

This example triggers an update whenever the hash of "my-function.zip" changes.

• https://github.com/hashicorp/terraform-provider-aws/issues/17989

Hi All,

We are on Terraform 0.14.6 and experiencing the following issue.
We are providing source_code_hash for the aws_lambda_layer_version in the plan terraform accepts it but writes totally different to the state file.

In the plan the source_code_hash is FyN0P9BvuTm023dkHFaWvAGmyD0rlhujGsPCTqaBGyw= however in the state file it becames c3forIEso3mJh74PY6HrhFK94GfJvQ4zG9rEIgBCBhw=.

When I check the layer in AWS CLI the "CodeSha256": c3forIEso3mJh74PY6HrhFK94GfJvQ4zG9rEIgBCBhw=,

Based on this it does not matter what kind of source_code_hash I can not overwrite hash of filename.

TF config.

  resource "aws_lambda_layer_version" "loader" {
  layer_name          = "loader"
  compatible_runtimes = ["python3.8"]

  filename         = "lambda_layer.zip"
  source_code_hash = filebase64sha256("lambda_layer.zip")
}

What you can see in all these examples, is that a hash is calculated to determine if the code has changed. That in itself isn’t an issue, but what is an issue, is that they all use a base64 encoded hash.

How to make your Lambda function deployment cross-environment friendly
The issue with base64 encoding, is that the resulting hash for the same data, will differ across environments (operating systems, user settings).

The following post describes this issue:

The root cause of is this is difference in packaging on different machines and bad documentation. Well, and an asinine design choice on AWS part.

source_code_hash gets overwritten by AWS-provided data upon response.
The documentation for source_code_hash (aka output_base64sha256, filebase64sha256) lies:

(String) The base64-encoded SHA256 checksum of output archive file.

Why would you even want to base64-encode a hash? The purpose of base64 encoding is to do away with non-printable chars, which a hash doesn’t have.

Turns out, what they actually do is compute sha256, then take the resulting text string and treat its characters as binary values, then base64 that: sha256sum lambda.zip | xxd -r -p | base64.

The problem is, recent zip versions store file permissions, and different umask values on different machines result in different permissions, which in turn produces different archives with different hashes.

But when you’re in a team where both Windows and macOS/Linux are being used, you have an additional challenge because the filesystems (and thus the filename of the archive-file) differ quite a lot.

Getting it to work

After some tinkering, I came to the following solution.

In my example, I supply the Lambda function code as a directory, containing the required file(s). In code, I create an archive file from that directory, using the data-source archive_file.

First, we create a random UUID, based on all the files (excluding ZIP-files) in the source directory (and child directories), and creating an MD5 hash for each of them.

# Create a random UUID which is used to trigger a redeploy of the function.
# The MD5 hash for each file (except ZIP-files) will be calculated and if any of those changes, 
# it will trigger a redeploy of the aws_lambda_function resource `lambda_function`.
# We cannot rely on a base64 hash, because the seed for that is environment dependent.
resource "random_uuid" "lambda_function" {
  keepers = {
    for filename in setunion(
      toset([for fn in fileset("${path.root}/lambda_function/", "**") : fn if !endswith(fn, ".zip")]),
    ) :
    filename => filemd5("${path.root}/lambda_function/${filename}")
  }
}

I chose to use MD5 here, because we’re not using it for cryptographic purposes. You can just as easily select SHA256 or SHA512, which do require some additional resources when calculating them (which might very well be negligible).

Next, we create a ZIP-file and send it to a different path than the source directory (which is excluded in .gitignore).

# Create an archive file of the function directory
data "archive_file" "lambda_function" {
  type        = "zip"
  source_dir  = "${path.root}/lambda_function"
  output_path = "${path.root}/lambda_output/${var.function_name}.zip"
}

When any of the source files changes, a new random UUID will be generated. To make sure this triggers a re-deployment of the Lambda function, we’ll set the random_uuid resource as a replacement trigger for the Lambda function resource.

To do this, we add a lifecycle-block to the aws_lambda_function resource, with a replace_triggered_by block, targeting the random_uuid.lambda resource.
The archive file will be created every time a terraform plan or terraform apply is run. Since the location of the resulting archive file will be different for every user (remember, we’re talking about dev/test deployments here!), we also need to make sure that the filename of the archive file doesn’t trigger unnecessary redeployments, by adding an ignore_changes block to the lifecycle block, targeting the filename property of the aws_lambda_function resource.

# Create the Lambda function
resource "aws_lambda_function" "lambda_function" {
  function_name = var.function_name
  role          = aws_iam_role.lambda_execution_role.arn
  handler       = "${var.function_name}.${var.handler_name}"
  runtime       = var.runtime
  timeout       = var.timeout
  architectures = var.architectures

  # Use the filename of the archive file as input for the function
  filename = data.archive_file.lambda_function.output_path

  depends_on = [
    aws_iam_role.lambda_execution_role
  ]

  lifecycle {
    replace_triggered_by = [
      # Trigger a replace of the function when any of the function source files changes.
      random_uuid.lambda_function
    ]
    ignore_changes = [
      # Ignore the source filename of the object itself, because that can change between 
      # users/machines/operating systems.
      filename
    ]
  }
}

Once this has been applied, when you now run the same code across different environments, no unexpected/undesired re-deployments of the Lambda function will occur.

This same approach can be used for deploying Lambda Layers. The difference there is that there’s an intermediate in the form of an S3 object, which will be replaced when there’s a change in any of the source files. Which, in turn, triggers replacing the Lambda Layer with a new version.

resource "random_uuid" "lambda_layer" {
  keepers = {
    for filename in setunion(
      toset([for fn in fileset("${path.root}/lambda_layer/", "**") : fn if !endswith(fn, ".zip")]),
    ) :
    filename => filemd5("${path.root}/lambda_layer/${filename}")
  }
}

data "archive_file" "lambda_layer" {
  type        = "zip"
  source_dir  = "${path.root}/lambda_layer"
  output_path = "${path.root}/lambda_output/${var.layer_name}.zip"
}

resource "aws_s3_object" "this" {
  depends_on         = [data.archive_file.lambda_layer]
  key                = join("/", [for x in [var.s3_key, join(".", [var.layer_name, "zip"])] : x if x != null && x != ""])
  bucket             = var.s3_bucket
  source             = data.archive_file.lambda_layer.output_path
  checksum_algorithm = "SHA256"

  lifecycle {
    replace_triggered_by = [
      random_uuid.lambda_layer
    ]
    ignore_changes = [
      # Ignore the source of the object itself, because that can change between machines/operating systems
      source
    ]
  }
}

resource "aws_lambda_layer_version" "lambda_layer" {
  layer_name          = var.layer_name
  compatible_runtimes = [var.runtime]
  source_code_hash    = aws_s3_object.this.checksum_sha256
  s3_bucket           = aws_s3_object.this.bucket
  s3_key              = aws_s3_object.this.key
}

When running all your IaC changes through a pipeline (as you should for at least production and staging/acceptance), this should not be an issue for you. But having your terraform plan/apply cluttered with false changes because of differences between contributor systems for your development and test-stages, should be in the past with this approach.

The Lambda module by Anton Babenko also uses base64 in its hash calculations, as well as the filename of the archive file. So if you run into (one of) the mentioned issues with that module, now you know why; I’ll be working on a PR to get the base64 part fixed for that module.

Conclusion

The goal of this post is to show you how to tackle (at least) two possible issues you might have with deploying Lambda functions and/or layers using Terraform.
I hope to have given you some insight into the causes of these issues, and with that, to make an informed decision on how to tackle them.