DEV Community: Sebastiaan Brozius

Consistently deploying Lambda functions and layers using Terraform

Sebastiaan Brozius — Mon, 29 Dec 2025 18:07:19 +0000

The code that accompanies this blogpost can be found here

Deploying Lambda functions to AWS using Terraform can be quite a struggle, especially when deploying from multiple environments (which only happens in dev and test environments, am I right?).

Some issues you can encounter are:

Lambda functions redeploying at every terraform apply
Errors about missing archive files containing the Lambda function files
Soft locks in the Terraform state file

In this post I’ll be showing you a way to be able to consistently deploy Lambda functions, only when there are changes to the code, when deploying from multiple environments.

History

There have been long time issues when deploying Lambda functions using Terraform. Some external links with examples of these issues and some attempts to tackle them:

https://stackoverflow.com/q/52662244/19024815

I have an AWS Lambda deployed successfully with Terraform:

resource "aws_lambda_function" "lambda" {
 filename                       = "dist/subscriber-lambda.zip"
 function_name                  = "test_get-code"
 role                           = <my_role>
 handler                        = "main.handler"
 timeout                        = 14
 reserved_concurrent_executions = 50
 memory_size                    = 128
 runtime                        = "python3.6"
 tags                           = <my map of tags>
 source_code_hash               = "${base64sha256(file("../modules/lambda/lambda-code/main.py"))}"
 kms_key_arn                    = <my_kms_arn>
 vpc_config {
   subnet_ids         = <my_list_of_private_subnets>
   security_group_ids = <my_list_of_security_groups>
 }
 environment {
   variables = {
     environment = "dev"
   }
 }
}

Now, when I run terraform plan command it says my lambda resource needs to be updated because the source_code_hash has changed, but I didn't update lambda Python codebase (which is versioned in a folder of the same repo):

 ~ module.app.module.lambda.aws_lambda_function.lambda
 last_modified:                     "2018-10-05T07:10:35.323+0000" => <computed>
 source_code_hash:                  "jd6U44lfe4124vR0VtyGiz45HFzDHCH7+yTBjvr400s=" => "JJIv/AQoPvpGIg01Ze/YRsteErqR0S6JsqKDNShz1w78"

https://nulldog.com/trigger-terraform-lambda-code-uploads-a-guide

Trigger updates with null_resource:

If you need more control or want to trigger updates based on other resources, use null_resource:
resource "null_resource" "lambda_update" {
  triggers = {
    code_hash = filebase64sha256("my-function.zip")
  }

  provisioner "local-exec" {
    command = "echo 'Code updated, triggering Lambda deployment...'"
  }
}

resource "aws_lambda_function" "example" {
  # ... other configurations
  depends_on = [null_resource.lambda_update]
}
This example triggers an update whenever the hash of "my-function.zip" changes.

https://github.com/hashicorp/terraform-provider-aws/issues/17989

Hi All,

We are on Terraform 0.14.6 and experiencing the following issue.
We are providing source_code_hash for the aws_lambda_layer_version in the plan terraform accepts it but writes totally different to the state file.

In the plan the source_code_hash is FyN0P9BvuTm023dkHFaWvAGmyD0rlhujGsPCTqaBGyw= however in the state file it becames c3forIEso3mJh74PY6HrhFK94GfJvQ4zG9rEIgBCBhw=.

When I check the layer in AWS CLI the "CodeSha256": c3forIEso3mJh74PY6HrhFK94GfJvQ4zG9rEIgBCBhw=,

Based on this it does not matter what kind of source_code_hash I can not overwrite hash of filename.

TF config.
  resource "aws_lambda_layer_version" "loader" {
  layer_name          = "loader"
  compatible_runtimes = ["python3.8"]

  filename         = "lambda_layer.zip"
  source_code_hash = filebase64sha256("lambda_layer.zip")
}

What you can see in all these examples, is that a hash is calculated to determine if the code has changed. That in itself isn’t an issue, but what is an issue, is that they all use a base64 encoded hash.

How to make your Lambda function deployment cross-environment friendly
The issue with base64 encoding, is that the resulting hash for the same data, will differ across environments (operating systems, user settings).

The following post describes this issue:

The root cause of is this is difference in packaging on different machines and bad documentation. Well, and an asinine design choice on AWS part.

source_code_hash gets overwritten by AWS-provided data upon response.
The documentation for source_code_hash (aka output_base64sha256, filebase64sha256) lies:

(String) The base64-encoded SHA256 checksum of output archive file.

Why would you even want to base64-encode a hash? The purpose of base64 encoding is to do away with non-printable chars, which a hash doesn’t have.

Turns out, what they actually do is compute sha256, then take the resulting text string and treat its characters as binary values, then base64 that: sha256sum lambda.zip | xxd -r -p | base64.

The problem is, recent zip versions store file permissions, and different umask values on different machines result in different permissions, which in turn produces different archives with different hashes.

But when you’re in a team where both Windows and macOS/Linux are being used, you have an additional challenge because the filesystems (and thus the filename of the archive-file) differ quite a lot.

Getting it to work

After some tinkering, I came to the following solution.

In my example, I supply the Lambda function code as a directory, containing the required file(s). In code, I create an archive file from that directory, using the data-source archive_file.

First, we create a random UUID, based on all the files (excluding ZIP-files) in the source directory (and child directories), and creating an MD5 hash for each of them.

# Create a random UUID which is used to trigger a redeploy of the function.
# The MD5 hash for each file (except ZIP-files) will be calculated and if any of those changes, 
# it will trigger a redeploy of the aws_lambda_function resource `lambda_function`.
# We cannot rely on a base64 hash, because the seed for that is environment dependent.
resource "random_uuid" "lambda_function" {
  keepers = {
    for filename in setunion(
      toset([for fn in fileset("${path.root}/lambda_function/", "**") : fn if !endswith(fn, ".zip")]),
    ) :
    filename => filemd5("${path.root}/lambda_function/${filename}")
  }
}

I chose to use MD5 here, because we’re not using it for cryptographic purposes. You can just as easily select SHA256 or SHA512, which do require some additional resources when calculating them (which might very well be negligible).

Next, we create a ZIP-file and send it to a different path than the source directory (which is excluded in .gitignore).

# Create an archive file of the function directory
data "archive_file" "lambda_function" {
  type        = "zip"
  source_dir  = "${path.root}/lambda_function"
  output_path = "${path.root}/lambda_output/${var.function_name}.zip"
}

When any of the source files changes, a new random UUID will be generated. To make sure this triggers a re-deployment of the Lambda function, we’ll set the random_uuid resource as a replacement trigger for the Lambda function resource.

To do this, we add a lifecycle-block to the aws_lambda_function resource, with a replace_triggered_by block, targeting the random_uuid.lambda resource.
The archive file will be created every time a terraform plan or terraform apply is run. Since the location of the resulting archive file will be different for every user (remember, we’re talking about dev/test deployments here!), we also need to make sure that the filename of the archive file doesn’t trigger unnecessary redeployments, by adding an ignore_changes block to the lifecycle block, targeting the filename property of the aws_lambda_function resource.

# Create the Lambda function
resource "aws_lambda_function" "lambda_function" {
  function_name = var.function_name
  role          = aws_iam_role.lambda_execution_role.arn
  handler       = "${var.function_name}.${var.handler_name}"
  runtime       = var.runtime
  timeout       = var.timeout
  architectures = var.architectures

  # Use the filename of the archive file as input for the function
  filename = data.archive_file.lambda_function.output_path

  depends_on = [
    aws_iam_role.lambda_execution_role
  ]

  lifecycle {
    replace_triggered_by = [
      # Trigger a replace of the function when any of the function source files changes.
      random_uuid.lambda_function
    ]
    ignore_changes = [
      # Ignore the source filename of the object itself, because that can change between 
      # users/machines/operating systems.
      filename
    ]
  }
}

Once this has been applied, when you now run the same code across different environments, no unexpected/undesired re-deployments of the Lambda function will occur.

This same approach can be used for deploying Lambda Layers. The difference there is that there’s an intermediate in the form of an S3 object, which will be replaced when there’s a change in any of the source files. Which, in turn, triggers replacing the Lambda Layer with a new version.

resource "random_uuid" "lambda_layer" {
  keepers = {
    for filename in setunion(
      toset([for fn in fileset("${path.root}/lambda_layer/", "**") : fn if !endswith(fn, ".zip")]),
    ) :
    filename => filemd5("${path.root}/lambda_layer/${filename}")
  }
}

data "archive_file" "lambda_layer" {
  type        = "zip"
  source_dir  = "${path.root}/lambda_layer"
  output_path = "${path.root}/lambda_output/${var.layer_name}.zip"
}

resource "aws_s3_object" "this" {
  depends_on         = [data.archive_file.lambda_layer]
  key                = join("/", [for x in [var.s3_key, join(".", [var.layer_name, "zip"])] : x if x != null && x != ""])
  bucket             = var.s3_bucket
  source             = data.archive_file.lambda_layer.output_path
  checksum_algorithm = "SHA256"

  lifecycle {
    replace_triggered_by = [
      random_uuid.lambda_layer
    ]
    ignore_changes = [
      # Ignore the source of the object itself, because that can change between machines/operating systems
      source
    ]
  }
}

resource "aws_lambda_layer_version" "lambda_layer" {
  layer_name          = var.layer_name
  compatible_runtimes = [var.runtime]
  source_code_hash    = aws_s3_object.this.checksum_sha256
  s3_bucket           = aws_s3_object.this.bucket
  s3_key              = aws_s3_object.this.key
}

When running all your IaC changes through a pipeline (as you should for at least production and staging/acceptance), this should not be an issue for you. But having your terraform plan/apply cluttered with false changes because of differences between contributor systems for your development and test-stages, should be in the past with this approach.

The Lambda module by Anton Babenko also uses base64 in its hash calculations, as well as the filename of the archive file. So if you run into (one of) the mentioned issues with that module, now you know why; I’ll be working on a PR to get the base64 part fixed for that module.

Conclusion

The goal of this post is to show you how to tackle (at least) two possible issues you might have with deploying Lambda functions and/or layers using Terraform.
I hope to have given you some insight into the causes of these issues, and with that, to make an informed decision on how to tackle them.

Setting up AWS IoT Core using Terraform

Sebastiaan Brozius — Sun, 29 Dec 2024 20:55:47 +0000

The code that accompanies this blogpost can be found here

I've been tinkering with AWS IoT Core this year, and wanted to put at least some of what I've found and done into a blog, so here it is.

AWS IoT Core is the AWS Internet-of-Things service:

AWS IoT provides the cloud services that connect your IoT devices to other devices and AWS cloud services. AWS IoT provides device software that can help you integrate your IoT devices into AWS IoT-based solutions. If your devices can connect to AWS IoT, AWS IoT can connect them to the cloud services that AWS provides.

Setting up an IoT environment within AWS is pretty easy, but I want to put it in code, so I can easily reproduce the environment I set up, and also be able to easily remove the configuration. My tool of choice is (still) Terraform.

We'll need to create the following resources:

IAM role used for registering new things
IoT policy for devices
IoT thing group (optional)
IoT thing type (optional)
Pre-provisioning Lambda (optional)
IoT fleet provisioning template (with optional pre-provisioning hook)
IoT policy for provisioning
Certificate for claim-based provisioning
IoT event configurations (optional)

I've included Terraform code in the accompanying GitHub repository.

IAM role for registering new things

When provisioning, the role used by IoT requires permissions to register new things. A managed policy AWSIoTThingsRegistration exists for this purpose, which should be assigned to a (new) role.

# Create the IoT provisioning IAM role
resource "aws_iam_role" "iot_fleet_provisioning" {
  name = join("-", [var.name, "fleet-provisioning-role"])

  assume_role_policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Principal" : {
          "Service" : "iot.amazonaws.com"
        },
        "Action" : "sts:AssumeRole"
      }
    ]
  })
}

# Attach the managed role for registering things to the provisioning role
resource "aws_iam_role_policy_attachment" "iot_fleet_provisioning" {
  role       = aws_iam_role.iot_fleet_provisioning.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSIoTThingsRegistration"
}

# Ensure that these (managed) policies are the only ones attached to the provisioning role on every apply
resource "aws_iam_role_policy_attachments_exclusive" "iot_fleet_provisioning" {
  role_name = aws_iam_role.iot_fleet_provisioning.name
  policy_arns = [
    "arn:aws:iam::aws:policy/service-role/AWSIoTThingsRegistration",
  ]
}

IoT policy for devices

What actions are allowed for a thing, will be defined in an IoT policy. A thing authenticates with IoT Core using a device-specific certificate. During provisioning, the policy will be assigned to the device-specific certificate, as defined in the fleet provisioning template.

Below policy allows

# Create a device policy
resource "aws_iot_policy" "iot_device" {
  name = join("-", [var.name, "device-policy"])

  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : [
          "iot:Publish",
          "iot:Receive"
        ],
        "Resource" : [
          "arn:aws:iot:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:topic/$${iot:Connection.Thing.ThingName}/*",
        ]
      },
      {
        "Effect" : "Allow",
        "Action" : "iot:Subscribe",
        "Resource" : [
          "arn:aws:iot:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:topicfilter/$aws/things/$${iot:Connection.Thing.ThingName}/shadow/*",
        ]
      },
      {
        "Condition" : {
          "Bool" : {
            "iot:Connection.Thing.IsAttached" : [
              "true"
            ]
          }
        },
        "Effect" : "Allow",
        "Action" : "iot:Connect",
        "Resource" : "*"
      }
    ]
  })
}

IoT thing group (optional)

Thing groups are optional, and can be used to group things together. There are static and dynamic thing groups. There's a limit of 100 dynamic groups per account, so if you've got a large environment with possibly a lot of groups, think ahead on whether or not you can use dynamic groups.

Dynamic thing groups are created from specific search queries in the registry. Search query parameters such as device connectivity, device shadow creation, and AWS IoT Device Defender violations data support this. Dynamic thing groups require fleet indexing enabled to index, search, and aggregate your devices' data.

Static thing groups allow you to manage several things at once by categorizing them into groups. Static thing groups contain a group of things that are managed by using the console, CLI, or the API.

In this example we're using a static group, which the new thing will be assigned to by the fleet provisioning template.

# Create a Thing group
resource "aws_iot_thing_group" "provisioning" {
  name = "Provisioning"
}

IoT thing type (optional)

Thing types allow you to store description and configuration information that is common to all things associated with the same thing type.

Although thing types are optional, their use makes it easier to discover things.

Things with a thing type can have up to 50 attributes.
Things without a thing type can have up to three attributes.
A thing can be associated with only one thing type.
There is no limit on the number of thing types you can create in your account.

In this example we're creating a single thing type, which is assigned to the thing by the fleet provisioning template.

# Create a Thing type
# The delete process of these is that they'll be deprecated first,
# and 5 minutes later they can be deleted.
resource "aws_iot_thing_type" "example" {
  name = "Example"

  properties {
    description = "Example"
    searchable_attributes = [
      # There's a maximum of 3 searchable attributes per Thing Type
      "environment",
      "license",
    ]
  }
}

Pre-provisioning Lambda (optional)

AWS recommends using pre-provisioning hook functions when creating provisioning templates to allow more control of which and how many devices your account onboards. Pre-provisioning hooks are Lambda functions that validate parameters passed from the device before allowing the device to be provisioned. This Lambda function must exist in your account before you provision a device because it's called every time a device sends a request through RegisterThing.

In this example we're deploying a simple Lambda-function used for pre-provisioning. No logic is built into the Lambda, but it shows that you can have a gatekeeper present in your provisioning process. You could, for example, check if the license-number provided by the thing for registering is valid, if the IP-address of the thing is as expected, if you've reached your maximum number of things you want to register, et cetera.

import json


def pre_provisioning_hook(event, context):
    print(event)

    # You can put code here to check if a device trying to connect
    # should be allowed or not, like checking if any of the provided
    # attributes are valid.
    # This function has to be able to respond within 5 seconds,
    # otherwise the provisioning request fails.
    # Reference: https://docs.aws.amazon.com/iot/latest/developerguide/pre-provisioning-hook.html

    # If you want to allow the device to connect to IoT Core, return this:
    # 'allowProvisioning': True

    # If you want to disallow the device to connect to IoT Core, return this:
    # 'allowProvisioning': False
    return {
        'allowProvisioning': True
    }

IoT fleet provisioning template (with optional pre-provisioning hook)

The fleet provisioning template is what ties together all the resources we created previously.

This template defines the following:

The parameters it expects to receive from a thing that's submitting itself for registration, to be used in the template
The resource required for the thing to communicate with IoT; a thing, a certificate and one or more policies.

This is where we assign the device policy to the thing, through the device-specific certificate which will be created for the thing. It will assign the thing to the initial thing group, and assign a thing type. By using multiple fleet provisioning templates, with different provisioning certificates, you can easily register different types of devices in your IoT environment, and assign device-specific attributes to them.

# Create the fleet provisioning template
resource "aws_iot_provisioning_template" "fleet" {
  name                  = join("-", [var.name, "fleet-provisioning-tpl"])
  description           = "Fleet provisioning template for ${var.name}"
  provisioning_role_arn = aws_iam_role.iot_fleet_provisioning.arn
  enabled               = true

  template_body = jsonencode({
    "DeviceConfiguration" : {},
    "Parameters" : {
      "License" : {
        "Type" : "String"
      },
      "AWS::IoT::Certificate::Id" : {
        "Type" : "String"
      }
    },
    "Resources" : {
      "policy" : {
        "Type" : "AWS::IoT::Policy",
        "Properties" : {
          "PolicyName" : aws_iot_policy.iot_device.name
        }
      },
      "certificate" : {
        "Type" : "AWS::IoT::Certificate",
        "Properties" : {
          "CertificateId" : {
            "Ref" : "AWS::IoT::Certificate::Id"
          },
          "Status" : "Active"
        }
      },
      "thing" : {
        "Type" : "AWS::IoT::Thing",
        "OverrideSettings" : {
          "AttributePayload" : "MERGE",
          "ThingGroups" : "REPLACE",
          "ThingTypeName" : "REPLACE"
        },
        "Properties" : {
          "AttributePayload" : {
            "license" : { "Ref" : "License" },
          },
          "ThingGroups" : [
            aws_iot_thing_group.provisioning.name
          ],
          "ThingTypeName" : aws_iot_thing_type.example.name,
          "ThingName" : {
            "Fn::Join" : [
              "-",
              [
                "iot",
                {
                  "Ref" : "License"
                }
              ]
            ]
          }
        }
      }
    }
  })

  pre_provisioning_hook {
    target_arn      = aws_lambda_function.iot_preprovisioning.arn
    payload_version = "2020-04-01"
  }
}

IoT policy for provisioning

The environment we're setting up, uses the 'provisioning with claim' provisioning method. This means we don't have to create device certificates in advance, but a new device will register itself using a generic provisioning certificate.

Because this certificate will be 'out in the wild', we want to restrict the permissions it provides as much as possible. This means the certificate should only be allowed to be used to register a new thing, and create a device-specific certificate for that thing. This is also why we want to add the pre-provisioning hook as a gatekeeper.

Below policy allows the thing to connect to IoT, subscribe to, publish too and receive from MQTT topics related to certificate creation and provisioning, specific to a provisioning template.

# Create the claims provisioning certificate policy
resource "aws_iot_policy" "provisioning" {
  name = join("-", [var.name, "claim-certificate-policy"])

  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : "iot:Connect",
        "Resource" : "*"
      },
      {
        "Effect" : "Allow",
        "Action" : [
          "iot:Publish",
          "iot:Receive"
        ],
        "Resource" : [
          "arn:aws:iot:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:topic/$aws/certificates/create/*",
          "arn:aws:iot:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:topic/$aws/provisioning-templates/${aws_iot_provisioning_template.fleet.name}/provision/*"
        ]
      },
      {
        "Effect" : "Allow",
        "Action" : "iot:Subscribe",
        "Resource" : [
          "arn:aws:iot:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:topicfilter/$aws/certificates/create/*",
          "arn:aws:iot:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:topicfilter/$aws/provisioning-templates/${aws_iot_provisioning_template.fleet.name}/provision/*"
        ]
      }
    ]
  })
}

Certificate for claim-based provisioning

As the provisioning certificate, we're going to use a self-signed certificate. Reasons for using a self-signed certificate:

The ability to set how long the certificate will be valid
Not having to set up a Certificate Authority
An AWS IoT-generated certificate doesn't have the proper allowed uses to connect to MQTT

For this example we're setting the lifetime of the certificate to 365 days. The lifetime that's right for your environment will depend on things like how often you'll update/deploy the application that includes the provisioning template, and how easy it is to update the certificate in that application.

The IoT policy for provisioning will be assigned to this certificate, to make sure it's not going to be used for any actions other than registering a new thing.

# Create a self-signed provisioning certificate
resource "tls_private_key" "provisioning" {
  algorithm = "RSA"
  rsa_bits  = 2048
}

resource "tls_self_signed_cert" "provisioning" {
  private_key_pem = tls_private_key.provisioning.private_key_pem

  subject {
    common_name = "IoT Provisioning"
  }

  validity_period_hours = 8760 # 365 days

  allowed_uses = [
    "key_encipherment",
    "digital_signature",
    "server_auth",
  ]
}

# Add the provisioning certificate and attach the provisioning policy
resource "aws_iot_certificate" "iot_fleet_provisioning" {
  certificate_pem = tls_self_signed_cert.provisioning.cert_pem
  active          = true
}

resource "aws_iot_policy_attachment" "iot_fleet_provisioning_certificate" {
  policy = aws_iot_policy.provisioning.name
  target = aws_iot_certificate.iot_fleet_provisioning.arn
}

IoT event configurations (optional)

If you want to be able to act on IoT events, those will need to be enabled.

Enabling these, facilitates these events being published to specific MQTT topics. These can be used in IoT rules, to trigger actions when specific events happen. They can also be used by things, as long as the device policy grants permissions to subscribe to and receive from those topics.

This example enables events related to thing creation, updates, and deletion.

# Manage events that will publish messages to MQTT topics.
# Reference: https://docs.aws.amazon.com/iot/latest/developerguide/iot-events.html#iot-events-enable
resource "aws_iot_event_configurations" "this" {
  event_configurations = {
    "THING"                  = true,
    "THING_GROUP"            = false,
    "THING_TYPE"             = false,
    "THING_GROUP_MEMBERSHIP" = false,
    "THING_GROUP_HIERARCHY"  = false,
    "THING_TYPE_ASSOCIATION" = false,
    "JOB"                    = false,
    "JOB_EXECUTION"          = false,
    "POLICY"                 = false,
    "CERTIFICATE"            = false,
    "CA_CERTIFICATE"         = false,
  }
}

IoT logging to CloudWatch (not included)

By default, AWS IoT Core doesn't log to CloudWatch. You can enable this in the console under Settings, or using the Terraform resource aws_iot_logging_options.
This will incur extra costs, so do keep an eye on that.

IaC caveats

Lack of resource support

Not all resources can be created using Terraform. Support for jobs and jobs templates is missing, for example, which are resources that really help to create a workflow for provisioning and staging your things.

These resources can be created using the AWS SDK, so a workaround for this shortcoming is to create a Lambda that performs the desired action in a dynamic environment, or use the SDK in a script, when your environment is more static.

A way to leverage a Lambda function when deploying your environment using Terraform, is the resource aws_lambda_invocation. This way, if you set the triggers correctly, the Lambda-function will be invoked when any of the triggers changes.

Another solution can be to use a step function to orchestrate the provisioning of a new thing, but might be a bit of overkill, depending on the size of your environment. Starting out with one or more (simple) Lambda functions and later on refactoring this to a step function is always an option.

Destroying your environment

When destroying the environment using Terraform, any device certificate needs to be deactivated, and detached from any policy (and thing). Otherwise the policies cannot be removed.

Sample client

Now that we've set up the IoT environment, it's time to test it. I've included a Python sample client in the accompanying GitHub repository, which registers itself with IoT Core, and writes the device-specific certificates to disk.

❯ python3 ./iotservice.py
Connecting to akgbiozgh01fa-ats.iot.eu-west-1.amazonaws.com with client ID 'iot-123'...
Lifecycle Connection Success
Connected!
Subscribing to CreateKeysAndCertificate Accepted topic...
Subscribing to CreateKeysAndCertificate Rejected topic...
Subscribing to RegisterThing Accepted topic...
Subscribing to RegisterThing Rejected topic...
Publishing to CreateKeysAndCertificate...
Waiting... CreateKeysAndCertificateResponse: null
Published CreateKeysAndCertificate request..
Received a new message awsiot.iotidentity.CreateKeysAndCertificateResponse(certificate_id='<CERTIFICATE_ID>', certificate_ownership_token='<CERTIFICATE_OWNERSHIP_TOKEN>', certificate_pem='<CERTIFICATE_PEM>', private_key='<PRIVATE_KEY>')
Publishing to RegisterThing topic...
Waiting... RegisterThingResponse: null
Published RegisterThing request..
Received a new message awsiot.iotidentity.RegisterThingResponse(device_configuration={}, thing_name='iot-123')
Exiting Sample: success
Stop the Client...
No Client to stop
Thing name: iot-123

This example is based on examples provided by AWS.

Device registration caveats

The device-specific certificates are written to disk in the function registerthing_execution_accepted in fleetprovisioning_mqtt5.py. IoT Core creates the device certificates before pre-provisioning has finished. When writing the certificates to disk, while the device is rejected by pre-provisioning, any later attempts to connect can fail, because there are already device-specific certificates present on the device. That would mean the certificates on the device need to be removed, before a new attempt can be made.

Also, because the certificate is created before the device is actually accepted, there will be certificates listed in IoT Core with the status Pending activation.

Conclusion

I've had fun this year figuring out things like this, and hope I've been able to provide you with enough information to set up your own IoT Core environment and play around with it.

Think ahead of the challenges you think you'll be facing, and be agile. Start small, and prepare for expanding to a larger scale. And as always, variables and requirements can (and probably will) change. Knowing what your options are, what the pros and cons are of those options will greatly help in picking the solutions you need both short, and long term.

Exporting an AMI to multiple formats

Sebastiaan Brozius — Sat, 07 Dec 2024 21:11:56 +0000

This is a short follow-up on my previous post.

In my previous post about creating an AMI with Image Builder, I exported that AMI to a different image format (vmdk).

There's two things I ran into with that solution:

You can only export to a single other image format
There is no apparent correlation between the exported image and the original AMI

To work around those two items, I created an ECS task definition, which uses a container with QEMU installed, as well as Python and Boto3. The entrypoint for the container is a Python script to handle the image conversion.

I set up the Image Builder pipeline without the export to S3, and have it send a notification to SNS.

The notification triggers a Lambda function (for which I used Python), which checks the status (event['Records'][0]['Sns']['Message']['state']['status']) of the pipeline run. This will be either FAILED or AVAILABLE.

The SNS message also contains the AMI ID (event['Records'][0]['Sns']['Message']['outputResources']['amis'][0]['image']), pipeline version (event['Records'][0]['Sns']['Message']['version']), build version (event['Records'][0]['Sns']['Message']['buildVersion']) and the name of the Image Builder pipeline (event['Records'][0]['Sns']['Message']['name']).

These are used by the Lambda function, to run an ECS task, using the pre-defined task definition with these values as environment variables for the ECS task.

The ECS task uses the AMI ID to start an export, which takes time. How much, depends on the AMI you're exporting. Since the export is initiated using Boto3, we get the export image task ID (ExportImageTaskId) in the response; this gives us the correlation we miss when we do the export using the distribution configuration of the pipeline. Once the export is done, you can pick up the exported image, and do additional conversions using the qemu-img command.

You can use the pipeline name, version and build version to name the images, so you can correlate them to the actual pipeline run that created the AMI you've created your export(s) from.

For exporting the images, the filesystem that's assigned to the Lambda function is used. In the task definition you can override the storage size using the EphemeralStorage setting.

When converting to multiple image formats, you might want to consider using multiprocessing. That way you can upload an image as soon as it's conversion is done, while also being able to start conversion for the next image format.

Once all the conversions are done, you could, once again, send out an SNS message to inform you of the status of the conversion process.

No code this time, but I did want to at least describe how you can run your custom process once the AMI is created.

Creating an AMI with Image Builder

Sebastiaan Brozius — Tue, 05 Nov 2024 23:37:13 +0000

The code that accompanies this blogpost can be found here

Update 2024-12-07: In the examples I use Amazon Linux 2023, which cannot be exported to other formats. I've changed the code in the repository to use an Ubuntu 24.04 base image, which can be exported.

I've been working with AWS Image Builder a lot more over the last couple of months, while replacing a Packer setup that was run on a Windows laptop, with Image Builder.

From the AWS Image Builder landing page:

EC2 Image Builder simplifies the building, testing, and deployment of Virtual Machine and container images for use on AWS or on-premises.

Keeping Virtual Machine and container images up-to-date can be time consuming, resource intensive, and error-prone. Currently, customers either manually update and snapshot VMs or have teams that build automation scripts to maintain images.

Image Builder significantly reduces the effort of keeping images up-to-date and secure by providing a simple graphical interface, built-in automation, and AWS-provided security settings. With Image Builder, there are no manual steps for updating an image nor do you have to build your own automation pipeline.

Image Builder is offered at no cost, other than the cost of the underlying AWS resources used to create, store, and share the images.

There are some caveats when using Image Builder:

EBS encryption by default should be off. An encrypted volume cannot be exported to an alternative image format.
The S3 bucket the exported image will be stored in, should use the AWS managed KMS key for S3 (SSE-S3) for encryption.
AWS encourages you to use IMDSv2 when running EC2 instances. This requires an adjustments to any scripts querying the instance metadata, as well as an adjustment to the maximum number of hops for an HTTP put request.

More information on these caveats can be found later in this post.

Creating an Image Builder pipeline

To create an Image Builder pipeline, the following resources are needed:

IAM roles with permissions for building the Amazon Machine Image (AMI), lifecycle management of the created AMIs, and for exporting the AMI to an additional image format
An S3 bucket to export the additional image format to
Any custom components for building your custom AMI
An image recipe
An infrastructure configuration
A distribution configuration
The Image Builder pipeline
(Optional) an SNS topic

In the GitHub repository I've linked, the code for the IAM-roles can be found in iam.tf, the code for the S3 bucket can be found in s3.tf, the code for SNS can be found in sns.tf, and the code for the remaining resources can be found in main.tf.

Make sure you're using at least version 5.74.0 of the Terraform AWS provider, to be able to enjoy these enhancements:

In version 5.74.0 support was added to the aws_imagebuilder_distribution_configuration resource for exporting the AMI to S3.
In version 5.59.0 support was added to the aws_imagebuilder_image_pipeline resource to set the workflow of the pipeline.

Custom components

A component can have multiple steps, in any of the two phases build ortest. It should have at least one step, and can contain steps for both build as well as test.

The first phase that is run, is the build phase. This is where the initial image if built. After the image has been created, a new EC2 instance (or container) will be started using that image, to run the test steps of the used components.

In the example, I'm using a simple component, which sets the timezone of the AMI to Europe/Amsterdam during the build phase.

resource "aws_imagebuilder_component" "set_timezone" {
  name         = join("-", [var.name, "set-timezone-linux"])
  description  = "Sets the timezone to Europe/Amsterdam"
  platform     = "Linux"
  version      = "1.0.0"
  skip_destroy = false # Setting this to true retains any previous versions
  data = yamlencode({
    schemaVersion = 1.0
    phases = [{
      name = "build"
      steps = [
        {
          name      = "SetTimezone"
          action    = "ExecuteBash"
          onFailure = "Abort"
          inputs = {
            commands = [
              "timedatectl set-timezone Europe/Amsterdam"
            ]
          }
        }
      ]
    }]
  })
}

Image recipe

The image recipe brings together all the 'ingredients' that make the image.

The recipe is where you define the source image (parent_image) you're building on, making overrides to settings of the source, as well as adding your own or AWS managed components, which are executed in the order they're listed in the recipe (per phase).

You can also have the SSM agent removed after building the image. This is useful to do if the image will be used outside of AWS, where the SSM agent has no use.

resource "aws_imagebuilder_image_recipe" "this" {
  # Currently the service only supports x86-based images for import or export.
  name         = join("-", [var.name, "image-recipe"])
  parent_image = "arn:aws:imagebuilder:eu-west-1:aws:image/amazon-linux-2023-ecs-optimized-x86/x.x.x"
  version      = "1.0.0"

  block_device_mapping {
    # The device name is the same device name as the root volume of the selected AMI,
    # which means we're overriding (some of) the root disk configuration in the AMI.
    # In this case we're increasing the size of the disk from 20 GB to 40 GB.
    device_name = "/dev/xvda"
    no_device   = false

    ebs {
      delete_on_termination = true
      volume_size           = 40
      volume_type           = "gp3"
      encrypted             = false
      iops                  = 3000
      throughput            = 125
    }
  }

  # Add the components to the recipe.
  # Recipes require a minimum of one build component, and can have a maximum of 20 build and test components in any combination.
  # Components are executed in the order they are listed here.
  component {
    # Here we're adding an AWS managed component to install the AWS CLI
    component_arn = "arn:aws:imagebuilder:${data.aws_region.current.name}:aws:component/aws-cli-version-2-linux/x.x.x"
  }

  component {
    # Here we're adding our custom component
    component_arn = aws_imagebuilder_component.set_timezone.arn
  }

  systems_manager_agent {
    # Set this to false to keep the SSM agent installed after building the image.
    uninstall_after_build = true
  }

  lifecycle {
    # Adding resources to the replace_triggered_by, ensures that replacing a resource doesn't fail because of dependencies.
    # Instead, this resource will be replaced as well.
    replace_triggered_by = [
      aws_imagebuilder_component.set_timezone
    ]
  }
}

Infrastructure configuration

The infrastructure configuration defines what instance type(s) can be used to build and test the image, which subnet the build/test instances should use, as well as which security group(s) should be attached to the instance. The instance profile to use is also defined here, as well as the SNS topic to send messages to upon either success or failure of the pipeline run.

If no subnet ID and security group IDs are provided, a subnet from the default VPC will be used, with the default security group. When providing a subnet ID, one or more security group IDs must also be provided.

If you run into issues during the build phase, you can set terminate_instance_on_failure to false. This means the build-instance will not be terminated, and can be used to investigate the issue.

In this example, IMDSv2 is used (http_tokens = required). Also see here for more information about the http_put_response_hop_limit.

resource "aws_imagebuilder_infrastructure_configuration" "this" {
  name                  = join("-", [var.name, "infrastructure-config"])
  description           = "Infrastructure Configuration for ${var.name}."
  instance_profile_name = aws_iam_instance_profile.imagebuilder_build.name
  instance_types        = var.instance_types
  sns_topic_arn         = aws_sns_topic.this.arn
  # If you want to keep the instance when an error occurs, so you can debug the issue, set this to false
  terminate_instance_on_failure = true
  # When not providing a subnet id and security group id(s),
  # Image Builder uses a subnet in the default VPC with the default security group.
  security_group_ids = var.security_group_ids
  subnet_id          = var.subnet_id

  instance_metadata_options {
    http_tokens                 = "required"
    http_put_response_hop_limit = 1 # Increase this to 3 when building a container image
  }

  tags = {
    ImageType = "CustomisedAmazonLinux2023Image"
  }
}

Distribution configuration

The distribution configuration tells Image Builder how to name the output AMI, and how to distribute the output AMI to different accounts, regions, organisations, and export the AMI to an alternative image format (VHD, VMDK or RAW)

resource "aws_imagebuilder_distribution_configuration" "this" {
  name        = join("-", [var.name, "distribution-config"])
  description = "Distribution Configuration for ${var.name}."

  distribution {
    region = data.aws_region.current.name
    ami_distribution_configuration {
      name       = join("-", [var.name, "{{ imagebuilder:buildDate }}-{{ imagebuilder:buildVersion }}"])
      kms_key_id = null
      ami_tags = {
        ImageType = "CustomisedAmazonLinux2023Image"
      }
    }
    s3_export_configuration {
      role_name         = aws_iam_role.vmexport.name
      disk_image_format = upper(var.image_export_format)
      s3_bucket         = aws_s3_bucket.this.id
    }
  }
}

Image Builder pipeline

The Image Builder pipeline is what ties all the previous resources together. This orchestrates building the image, and additionally trigger scanning of the output AMI for security issues. Amazon Inspector should be enabled in the account to be able to scan the image.

The pipeline also defines the workflow to use. By default, a workflow that runs both the build and the test phases is used. In the example, no test components are used, so we're shaving some time off of the pipeline runtime, by selecting an AWS-managed workflow that only runs the build phase. When changing the default workflow, an execution role must also be provided.

The pipeline can also be scheduled to run at certain intervals using cron expressions. The pipeline can also be triggered using EventBridge rules, which requires additional resources to set up, which are not included in this sample.

When no schedule is provided, the pipeline can only be run manually, or when targeted by an EventBridge rule.

resource "aws_imagebuilder_image_pipeline" "this" {
  name                             = join("-", [var.name, "image-pipeline"])
  description                      = "Pipeline to create the custom image for ${var.name}"
  image_recipe_arn                 = aws_imagebuilder_image_recipe.this.arn
  infrastructure_configuration_arn = aws_imagebuilder_infrastructure_configuration.this.arn
  distribution_configuration_arn   = aws_imagebuilder_distribution_configuration.this.arn

  image_scanning_configuration {
    # Amazon Inspector needs to be enabled for the account when setting this to true
    image_scanning_enabled = false
  }

  image_tests_configuration {
    image_tests_enabled = true
    timeout_minutes     = 720
  }

  # When changing the workflow from default, an execution role must also be provided
  execution_role = "arn:aws:iam::${data.aws_caller_identity.account.account_id}:role/aws-service-role/imagebuilder.amazonaws.com/AWSServiceRoleForImageBuilder"
  workflow {
    # We're setting an AWS managed workflow, that only executes Build-steps of the component. No testing or validation is done.
    workflow_arn = "arn:aws:imagebuilder:${data.aws_region.current.name}:aws:workflow/build/build-image/x.x.x"
  }

  # Here you can set one or more schedules, to automate image building.
  dynamic "schedule" {
    for_each = var.schedule_expression != null ? [1] : []
    content {
      schedule_expression = var.schedule_expression
    }
  }

  lifecycle {
    # Adding resources to the replace_triggered_by, ensures that replacing a resource doesn't fail because of dependencies.
    # Instead, this resource will be replaced as well.
    replace_triggered_by = [
      aws_imagebuilder_image_recipe.this
    ]
  }
}

More info on the caveats of using Image Builder

EBS Encryption by default

To check if EBS encryption by default is enabled, we can use the following AWS CLI command:

$ aws ec2 get-ebs-encryption-by-default

{
    "EbsEncryptionByDefault": false
}

If it's true, check for Service Control Policies (SCPs) and/or other tooling used for managing the organisation/accounts. If this feature has been enabled through an SCP or other automated way, disable it for the account you'll be running Image Builder in.

To manually disable EBS encryption by default, the following AWS CLI command can be used:

$ aws ec2 disable-ebs-encryption-by-default

{
    "EbsEncryptionByDefault": false
}

{{< alert cardColor="orange" iconColor="#1d3557" textColor="#f1faee" >}}
If EBS encryption by default has been set using an SCP or other tooling, manually disabling will not be permanent, or might not even take effect.
{{< /alert >}}

S3 bucket encryption

To be able to export the image to an S3 bucket, the bucket needs to be encrypted with the default AWS managed KMS key for S3 (SSE-S3). Otherwise the export will fail with an InsufficientPermissions exception.

Instance Metadata Service (IMDS)

AWS encourages the use of IMDSv2 over IMDSv1. Version 2 is more secure, but requires some adjustments to any existing script using IMDSv1 when querying the instance metadata.

IMDSv1:

curl http://169.254.169.254/

IMDSv2:

TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/

Basic check to see if IMDSv1 or IMDSv2 is enabled:

if [ -z $(curl -s http://169.254.169.254/) ]; then echo "Instance has been configured to use IMDSv2."; fi

More information can be found here

Another setting that will need adjustment when using Image Builder for building a container image, is setting the Metadata Hop Limit (HttpPutResponseHopLimit) to 2 or 3.

More information on the IMDS options can be found here.

Conclusion

My goal with this post was to show you how you can start using Image Builder to automate creating your custom AMIs or container images, and help you take that initial hurdle to start looking into Image Builder.

It also shows the issues I ran into while implementing Image Builder for a project I'm working on, and how to overcome those.

If you have any feedback on this post, please reach out to me.

Dedicated game server and AWS Instance Scheduler

Sebastiaan Brozius — Wed, 20 Dec 2023 11:17:01 +0000

Recently I've started playing a game (Satisfactory, for those who want to know) with my two kids. To make it easier to play together, I've set up a dedicated server on an EC2 instance so the game is also available when the host is not available.

To keep the costs down a bit, I was looking for a way to shut down the instance during the hours we wouldn't be playing anyway. That's when I came across Instance Scheduler on AWS.

The Instance Scheduler on AWS solution automates the starting and stopping of Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Relational Database Service (Amazon RDS) instances.

This solution helps reduce operational costs by stopping resources that are not in use and starting resources when their capacity is needed. For example, a company can use Instance Scheduler on AWS in a production environment to automatically stop instances outside of business hours every day. If you leave all of your instances running at full utilization, this solution can result in up to 70% cost savings for those instances that are only necessary during regular business hours (weekly utilization reduced from 168 hours to 50 hours).

Instance Scheduler on AWS leverages Amazon Web Services (AWS) resource tags and AWS Lambda to automatically stop and restart instances across multiple AWS Regions and accounts on a customer-defined schedule. This solution also allows you to use hibernation for stopped Amazon EC2 instances.

This solution can be deployed in a single account, or in a central account, managing multiple accounts. Since I'm using an AWS Organization, I've deployed the main stack in a hub account, and the remote stack in the account that hosts the dedicated server instance.

To deploy Instance Scheduler (version 1.5.3 at the time of writing), go to the AWS CloudFormation console in your desired account and create a new stack using new resources. Paste the link to the CloudFormation Instance Scheduler template, as linked on the solution page, in the Amazon S3 URL field and click Next.

On the next page, you can configure the solution. Give the stack a meaningful name, and set the options you need. For mu configuration, I've set the default timezone to Europe/Amsterdam, set the option This account to No since I won't be hosting resources in the hub account, set the Namespace, and set Use AWS Organizations to Yes. When using AWS Organizations, provide the organization ID in the Organization Id/Remote Account Ids field, otherwise provide a comma separated list of account IDs you want to manage through this account. If you forget to provide a value for this, you will get an error on the schedulereventbuspolicy resource during deployment.

If you want to manage resources in other regions than where you're deploying the stack in, also make sure to enter those in the Region(s) field.

When you're done with this initial configuration, click on Next, review the settings on the Configure stack options page and click Next again.

Do a last review of the configuration, don't forget to tick the box to acknowledge that AWS CloudFormation might create IAM resources, and click on Submit.

The stack will create the resources for the solution, which takes a couple of minutes.

Once the resources have been deployed, we can create a schedule for the server.

The schedule is configured in DynamoDB. The CloudFormation stack created several DynamoDB tables. The schedules and periods used on the schedule are in a table names InstanceScheduler-main-ConfigTable-XXXXXXXXXXXX.

I've created the following periods and schedule in the table.

Periods used:

Workdays period:

{
  "type": {
    "S": "period"
  },
  "name": {
    "S": "satisfactory-workdays"
  },
  "begintime": {
    "S": "17:00"
  },
  "description": {
    "S": "Satisfactory Dedicated Server - workdays "
  },
  "endtime": {
    "S": "23:59"
  },
  "weekdays": {
    "SS": [
      "mon-thu"
    ]
  }
}

Weekends period 'on':

{
  "type": {
    "S": "period"
  },
  "name": {
    "S": "satisfactory-weekends-on"
  },
  "begintime": {
    "S": "12:00"
  },
  "description": {
    "S": "Satisfactory Dedicated Server - weekends"
  },
  "endtime": {
    "S": "23:59"
  },
  "weekdays": {
    "SS": [
      "fri-sun"
    ]
  }
}

Weekends period 'off':

{
  "type": {
    "S": "period"
  },
  "name": {
    "S": "satisfactory-weekends-off"
  },
  "begintime": {
    "S": "00:00"
  },
  "description": {
    "S": "Satisfactory Dedicated Server - weekends"
  },
  "endtime": {
    "S": "02:00"
  },
  "weekdays": {
    "SS": [
      "sat-sun"
    ]
  }
}

These periods turn the server on on weekdays between 17:00 and midnight, and for the weekends, turn it on on Fridays, Saturdays and Sundays at noon, until 02:00 on Saturdays and Sundays, and until midnight on Sundays, when combined into a schedule:

{
  "type": {
    "S": "schedule"
  },
  "name": {
    "S": "satisfactory"
  },
  "description": {
    "S": "Schedule for Satisfactory Dedicated Server"
  },
  "periods": {
    "SS": [
      "satisfactory-weekends-off",
      "satisfactory-weekends-on",
      "satisfactory-workdays"
    ]
  }
}

Once that's done, switch over to the account and region the dedicated server is in, and deploy the CloudFormation instance schedule remote template in CloudFormation.

In the stack details, specify the same Namespace used in the stack in the hub account, provide the account ID of the hub account in Hub Account ID and set Use AWS Organizations to Yes because that's what I'm using in the hub account.

When that stack has deployed, the server needs a tag so Instance Scheduler knows to manage this instance, and what schedule to use.
The tag we need to set on the EC2 instance is Schedule, with the value of the schedule we created, satisfactory.

And that's that!

An alternative AWS provides is Resource Scheduler. With Resource Scheduler, you can 'only' schedule EC2 instances, and they are not checked for the desired state, but might fit your use case better. I chose for Instance Scheduler to see how it works, and because I like solutions that can be managed from a hub account.

For this post, I've written about a personal dedicated game server, a fun project. But in environments where you don't need your resources running all day (like test and dev) or even in production environments where you need to have an instance available only at specific times, this can be helpful to reduce costs.

Automating patching with AWS Systems Manager

Sebastiaan Brozius — Mon, 13 Nov 2023 07:00:00 +0000

The code that accompanies this blogpost can be found here

Recently I've been looking into patching Windows servers that have dependencies between them, using AWS Systems Manager.

The use-case was an application that exists of web servers, middleware servers and a database server.

The web servers have connections open to the database server, and the middleware servers run processes that get information from the database server.

The servers were patched manually, by stopping the services on the web servers and middleware servers first and checking that all middleware services were stopped, before stopping the databases. Once that was done, the servers were updated. After patching, the databases were first brought back online, before starting the middleware services and the web services again.

To set this up, I created some PowerShell scripts (with a little bit of SSM variable flavour) to be run on the instances to stop and start the services, as well as checking the services before continuing to the next step. These scripts were put as SSM documents, to be called from an automation document.

Example script (Start-Components.ps1):

try {
  $_serverRole = "{{ServerRole}}" # This is an SSM variable reference
  $_fqdn = "$((Get-WmiObject Win32_ComputerSystem).DNSHostName).$((Get-WmiObject Win32_ComputerSystem).Domain)"
  Write-Output "[INF] Starting Components on $($_fqdn) with server role '$_serverRole'"

  switch ($_serverRole) {
    Web { 
      Write-Output "[INF] Setting Startup Type for web services where the current StartType is Manual to Automatic and starting them."

      Get-Service iisadmin | Where-Object StartType -eq "Manual" | Set-Service -StartupType Automatic -Status Running
      Get-Service w3svc | Where-Object StartType -eq "Manual" | Set-Service -StartupType Automatic -Status Running
    }
    Middleware {  
      Write-Output "[INF] Doing stuff to enable the middleware services to start."

      # Your code here
    }
    Database {  
      Write-Output "[INF] Setting Startup Type for all database services where the current StartType is Manual to Automatic and starting them."
      Get-Date -Format "yyyy-MM-dd HH:mm:ss"

      Get-Service *sql* | Where-Object StartType -eq "Manual" | Set-Service -StartupType Automatic -Status Running

      Write-Output "[INF] Making sure all database services are started before continuing."
      # When there are no services that match the name, the while loop will not be entered.
      while (Get-Service *sql* | Where-Object Status -ne Running) {
        Write-Output "[DEB] [$(Get-Date -Format "yyyy-MM-dd HH:mm:ss")] Not all database services have started yet. Waiting a little longer."
        Start-Sleep -Seconds 60
      }
    }
    Default { }
  }
}
catch {
  Write-Output "[ERR] Failed to start components!"
  Write-Error $Error[0] -ErrorAction Continue
  exit 1
}

Which is consumed to create an SSM document using Terraform:

resource "aws_ssm_document" "patching_start_components" {
  name          = "Patching-StartComponents"
  document_type = "Command"
  target_type   = "/AWS::EC2::Instance"

  content = jsonencode({
    schemaVersion = "2.2"
    description   = "Patching Post-install Start Components Document"
    parameters = {
      ServerRole = {
        type        = "String"
        description = "Role of the server (Web, Middleware, Database, None)"
        default     = "None"
        allowedValues = [
          "Web",
          "Middleware",
          "Database",
          "None",
        ]
      }
    }
    mainSteps = [
      {
        action = "aws:runPowerShellScript"
        name   = "StartComponents"
        precondition = {
          StringEquals = [
            "platformType",
            "Windows"
          ]
        }
        inputs = {
          runCommand = split("\n", file("${path.cwd}/powershell_scripts/Start-Components.ps1"))
        }
      }
    ]
  })
}

Using an automation document, we can orchestrate the flow of patching. In the example code, I've also included a method to patch servers of the same function at different times. For this, the option PatchWindow has been added, with allowed values Monday and Wednesday. The output of each step is redirected to an encrypted CloudWatch log-group.

resource "aws_ssm_document" "patching_automation" {
  name            = "Patching-Automation"
  document_type   = "Automation"
  document_format = "YAML"

  content = <<EOT
description: |-
  # Patching Automation

  This script provides a staged patching experience. Services are stopped in a specific order on specific instances after which patching is run, and services are started again on servers in reverse order.
schemaVersion: '0.3'
parameters:
  PatchWindow:
    type: String
    allowedValues:
      - Monday
      - Wednesday
    description: Patch-window to run for. Determines which servers are affected.
mainSteps:
  - name: StopWebServerServices
    action: 'aws:runCommand'
    inputs:
      DocumentName: ${aws_ssm_document.patching_stop_components.name}
      Targets:
        - Key: 'tag:ServerRole'
          Values:
            - Web
        - Key: 'tag:PatchWindow'
          Values:
            - '{{PatchWindow}}'
      Parameters:
        ServerRole: Web
      CloudWatchOutputConfig:
        CloudWatchLogGroupName: ${aws_cloudwatch_log_group.automated_patching.name}
        CloudWatchOutputEnabled: true
    description: Stop the services on the web servers
    nextStep: StopMiddlewareServices
    onFailure: 'step:StartWebServerServices'
  - name: StopMiddlewareServices
    action: 'aws:runCommand'
    inputs:
      DocumentName: ${aws_ssm_document.patching_stop_components.name}
      Targets:
        - Key: 'tag:ServerRole'
          Values:
            - Middleware
        - Key: 'tag:PatchWindow'
          Values:
            - '{{PatchWindow}}'
      Parameters:
        ServerRole: Middleware
      CloudWatchOutputConfig:
        CloudWatchLogGroupName: ${aws_cloudwatch_log_group.automated_patching.name}
        CloudWatchOutputEnabled: true
    description: Stop the services on the middleware servers
    nextStep: StopDatabaseServices
    onFailure: 'step:StartMiddlewareServices'
  - name: StopDatabaseServices
    action: 'aws:runCommand'
    inputs:
      DocumentName: ${aws_ssm_document.patching_stop_components.name}
      Targets:
        - Key: 'tag:ServerRole'
          Values:
            - Database
        - Key: 'tag:PatchWindow'
          Values:
            - '{{PatchWindow}}'
      Parameters:
        ServerRole: Database
      CloudWatchOutputConfig:
        CloudWatchLogGroupName: ${aws_cloudwatch_log_group.automated_patching.name}
        CloudWatchOutputEnabled: true
    description: Stop the services on the database servers
    nextStep: PatchServers
    onFailure: 'step:StartDatabaseServices'
  - name: PatchServers
    action: 'aws:runCommand'
    inputs:
      DocumentName: AWS-RunPatchBaseline
      Targets:
        # Uncomment the following lines to only patch specific server-roles
        # - Key: 'tag:ServerRole'
        #   Values:
        #     - Web
        #     - Middleware
        #     - Database
        - Key: 'tag:PatchWindow'
          Values:
            - '{{PatchWindow}}'
      Parameters:
        Operation: Install
        RebootOption: RebootIfNeeded
      CloudWatchOutputConfig:
        CloudWatchLogGroupName: ${aws_cloudwatch_log_group.automated_patching.name}
        CloudWatchOutputEnabled: true
    description: Patch the servers
    nextStep: StartDatabaseServices
    onFailure: Abort
  - name: StartDatabaseServices
    action: 'aws:runCommand'
    inputs:
      DocumentName: ${aws_ssm_document.patching_start_components.name}
      Targets:
        - Key: 'tag:ServerRole'
          Values:
            - Database
        - Key: 'tag:PatchWindow'
          Values:
            - '{{PatchWindow}}'
      Parameters:
        ServerRole: Database
      CloudWatchOutputConfig:
        CloudWatchLogGroupName: ${aws_cloudwatch_log_group.automated_patching.name}
        CloudWatchOutputEnabled: true
    description: Start the services on the database servers
    nextStep: StartMiddlewareServices
    onFailure: Abort
  - name: StartMiddlewareServices
    action: 'aws:runCommand'
    inputs:
      DocumentName: ${aws_ssm_document.patching_start_components.name}
      Targets:
        - Key: 'tag:ServerRole'
          Values:
            - Middleware
        - Key: 'tag:PatchWindow'
          Values:
            - '{{PatchWindow}}'
      Parameters:
        ServerRole: Middleware
      CloudWatchOutputConfig:
        CloudWatchLogGroupName: ${aws_cloudwatch_log_group.automated_patching.name}
        CloudWatchOutputEnabled: true
    description: Start the services on the middleware servers
    nextStep: StartWebServerServices
    onFailure: Abort
  - name: StartWebServerServices
    action: 'aws:runCommand'
    inputs:
      DocumentName: ${aws_ssm_document.patching_start_components.name}
      Targets:
        - Key: 'tag:ServerRole'
          Values:
            - Web
        - Key: 'tag:PatchWindow'
          Values:
            - '{{PatchWindow}}'
      Parameters:
        ServerRole: Web
      CloudWatchOutputConfig:
        CloudWatchLogGroupName: ${aws_cloudwatch_log_group.automated_patching.name}
        CloudWatchOutputEnabled: true
    description: Start the services on the web servers
    isEnd: true
EOT
}

The automation document allows for some error-handling as well. As you can see in the example, when the step StopMiddlewareServices fails, it will skip to step StartMiddlewareServices (defined with the line onFailure: 'step:StartMiddlewareServices') and will proceed from there.

Once we have the automation document in place, we can create maintenance windows with an associated task, to execute the automation document for that triggers automatically executing the automation document.

resource "aws_ssm_maintenance_window" "install_window_monday" {
  enabled  = true
  name     = "patch-window-monday"
  schedule = local.patching.cron_patching_monday
  duration = 4
  cutoff   = 2
}

resource "aws_ssm_maintenance_window_task" "task_install_patches_monday" {
  window_id = aws_ssm_maintenance_window.install_window_monday.id
  name      = "install-patches-monday"
  task_type = "AUTOMATION"
  task_arn  = aws_ssm_document.patching_automation.name
  priority  = 5

  task_invocation_parameters {
    automation_parameters {
      document_version = "$LATEST"

      parameter {
        name   = "PatchWindow"
        values = ["Monday"]
      }
    }
  }
}

In this example, instances with a tag PatchWindow with a value of Monday will be targeted for the maintenance task.

After applying the code to your environment, instances can be included by setting two tags on them.
PatchWindow determines the maintenance window the instance will be included in. In this example, valid values are Monday and Wednesday.
ServerRole determines which actions in the PowerShell scripts will be taken. In this example, valid values are Web, Middleware, Database or None.

Lessons learned from migrating 42 servers to AWS

Sebastiaan Brozius — Mon, 05 Jun 2023 15:33:00 +0000

Earlier this year I've worked on a project where we had to migrate 42 servers from a data center to AWS as part of an AWS Migration Acceleration Program (MAP) deal. There was some pressure, since the contract with the data center would end about 2 months later and the customer didn't want to renew or extend the current contract with the data center.

During this project I learned some things I think are valuable and would like to share. This might save you some time and/or frustration when you're working on a similar project.

Disclaimer: this is not meant to be a full migration guide, but rather things I encountered and want to share. Take from this what you can use, and create a migration plan for your specific situation.

The tools used

For this project, the following tools were used:

AWS Migration Hub
AWS Application Migration Service
Terraform (or any IaC tool of your choice)
Scripting-language of your choice (I used PowerShell)
Bash-scripting (the servers were running Linux)

Step one: Inventarisation

To know what exactly you're dealing with, it's important to make an inventarisation of the environment you'll be migrating, and everything it makes use of or is used by. Having a clear overview makes it easier to spot potential issues, plan ahead and help set up a realistic time frame for the migration.

Network / subnets

Get all information about the current networks and subnets the servers are using, are the servers using static or dynamic IP addresses, are there any VPN connections, do we need to take any allow-listings (both inbound and outbound) into account and what public IP addresses are in use, if any. The customer might even have their own public IP range which they want to (partially) move to AWS.

Firewall rules

Try to get a complete overview of all firewall rules that are in place. This helps in determining which servers should be accessible from where, should be able to connect to where, and might also help you spot issues with the current setup so you can mitigate those in the new setup.

Traffic flows between servers

The firewall rules might be of use for that, but traffic between private subnets is often unrestricted. Knowing the traffic flows between servers will help you set up more restrictive (and safer) security groups within AWS.

DNS domains

Does the customer want to move DNS domains to AWS? Are there DNS-records that point to the servers and need to be changed? If there are changes to be made to DNS records and/or domains, some planning needs to be done to make sure that during the actual migration, you don't have to wait hours or days before a change has been propagated throughout the internet.

Certificates

Do the servers or any applications running on them use any certificate and how are they managed? Knowing this can help determine how to expose an application to the internet; can you use an Application Load Balancer or is the certificate managed through an automated system which runs on the server itself? In the latter case, you either have to use a Network Load Balancer, connect the server 'directly' to the internet (bad practice!) or change how the certificate is managed, which might have a big impact on the work that needs to be done.

Backup RTO and RPO

For setting up the new environment, it's important to know what RPO and RTO the customer requires for which service / application / server. Setting up AWS Backup in advance makes it easier to enable it during or just after the migration.

Software license requirements

Some software vendors use the MAC-address of a server to bind their licenses to. If that's the case, some additional actions need to be taken to ensure you don't have to keep changing your license registration with such vendors.

OS versions being used and patch level

Before you start working on preparing for the migration, it's wise to know which OSes are being used, and what versions. When you encounter older OS versions, there might be more work involved installing the necessary agents, or it might even be impossible. Also, knowing the current patch-level of the OSes and how patching is managed is important to know for the new environment in AWS.

Software used

Specific products and/or versions are eligible for additional discounts in AWS MAP, like commercial databases, SAP and more.

Step two: Make a plan

Once you've got at least most of the information, it's time to start making a plan.

Make an IP-plan for the new environment

You won't always be able to keep the current IP-addresses and creating a new IP-plan is important for setting up the new network and subnets, with proper sizing.

Determine the security groups to create

Make sure you know the needed security groups to allow traffic between servers, and allow the required inbound and outbound traffic. Determine which security groups to make and what servers to attach them to.

Make sure you have your MAP tag number (MPE ID)

This is needed when deploying resources to get the discount. Determine how you will apply the tags and what resources might need alternate tags.

Determine the use of a launch template (highly recommended!)

AWS Application Migration Service makes use of launch templates. Determining if you're going to use it and what settings you want to specify in it, helps to gather possible additional info.

Determine the order of migration

Make an initial order of migration, and keep validating that order until the actual migration.

Step three: Preparations

Once you've created an initial plan to work from, it's time to start the actual preparations.

Set up the the management account listed in the Migration Plan (the management account that's part of the MAP deal) and activate the Cost Allocation Tag required for MAP 2.0. This info should be made available to you by your AWS representative for the MAP deal.
Set up AWS Migration Hub and either install the Discovery Agent on the servers you're migrating, or, if you have access to the hypervisor layer, install the appliance. More info on these can be found in the AWS Documentation for AWS Migration Hub
Next, deploy the infrastructure you're migrating the servers to, including an initial security group to assign to the launch template, and an Instance Profile with the appropriate permissions. Next, create VPC Endpoints for the services SSM uses; this way you should be able to access the servers even when they cannot connect to the internet (which is probably the case during the test phases).

Also make sure that you tag everything with the map-migrated tag, also for the infrastructure. With Terraform, you can set this using the AWS provider parameter default_tags. More info on the exact tag value should be available through the AWS MAP channel.

Once you've got the basic infrastructure set up, initialise the AWS Application Migration Service. During the initialisation, also make sure to set up the default launch template. Make sure the MAP tag is added with the appropriate value, the security group, instance profile, network subnet, et cetera.
After AWS Application Migration Service has been initialised, the Replication Agent can be installed on the servers to be migrated. The replication agent uses TCP port 1500 to connect to the AWS Application Migration Service, so make sure any firewall allows TCP port 1500 outbound for the source servers.

NB: The Replication Agent requires the Linux headers to install. For older Linux-versions this could mean you have to locate the Linux headers for the specific release, since they might no longer be available through the distributions update manager.

Install the AWS SSM Agent on the source servers. This is helpful to be able to connect to the server through the AWS Console using Session Manager, or even using the Session Manager Plugin for the AWS CLI. During testing and the actual migration, this can prove useful when you're running into issues with any server.
Once the servers have started replicating, we have to play the waiting game.

Until the servers have finished their initial replication, there's not much to be done. The time it takes before the initial replication is done, is dependent on the speed of the internet connection, the total amount of data to be replicated, as well as the number of changes to the filesystem the source systems have. The AWS Application Migration Service console gives an estimate of the time required to complete replication, which is constantly updated.

Step four: Test, test, test!

When all (or at least the ones you want to start with) servers have replicated, you can start testing.

In the AWS Application Migration Service console, select one or more servers to test, and launch test instances.

Make sure that the test-servers are unable to contact live servers, so they do not contaminate any production environment.
Create a migration-script per server. Do multiple test-runs to check and improve the migration-scripts.
During testing, you might encounter software that can throw a wrench in the migration, like corosync and pacemaker. When you encounter such software, determine if you still need it and take action accordingly to mitigate any possible issues that might arise by keeping those configurations as they are.
Evaluate of your intended order of migration is valid. During testing you might find a different order is needed.
Create waves based on the order of migration for a simpler orchestration during the actual migration.
Do at least one full test-migration. This helps determine how much time is needed for the full migration. This is important for how much down-time you'll have, which needs to be communicated with the customer and any users of the application(s), as well as help in deciding the moment of the actual migration, the number of people working on the migration, when the test-persons should be able to start testing the application after migration, et cetera.
If you're moving any server from being directly exposed to the internet, to being fronted by a load balancer, test the load balancer configuration as best you can.
Once you're done testing a source server, mark it as 'Ready for cut-over' in AWS Application Migration Service.
If there are DNS changes to be made, prepare for them; lower TTL values for records that need to be changed, and prepare any domain that needs to be moved to Route53, or even move them in advance if possible.

Step five: The real deal

This is what you've been testing for!

Shut down any running services on the live servers, especially databases, and wait for the last changes to be replicated to AWS.
Start migrating in waves.
Make sure your security groups have the proper access (they should at least be reachable for the group of test-users)
Have your test-group test as early as possible and have a select group of people report on any findings. Triage what needs to be fixed right away, and what can wait. Have product owners participate in this where possible.
Mark servers that have been given the green light as 'Finalize cut-over' in AWS Application Migration Service to indicate they're finished.
Turn on VPC Flow Logs to help troubleshoot any network-issues during the migration.

Step six: The aftermath

Once the migration has been finished successfully, there's a few more things that need to be done.

Turn off the old servers, or at the very least make sure that the applications will not be enabled again.
Make sure the servers and services are being backed up in AWS.
Mark the migrated servers as 'Mark as archived' in AWS Application Migration Service.
Remove any software from the servers that was specifically needed for the data center architecture (e.g. VMware tool, Azure tools)

Points of attention

During both testing and the actual migration, when launching multiple (bigger) instances at the same time, one or more instances might respond badly/have weird issues. In that case, stop the instance(s) in the AWS Console, wait a minute or two, and start it up again. The reason for this is that the underlying host has issues allocating the proper resources to the instance. Stopping the instance and starting it again relocates the instance to a host that has sufficient resources available for the instance.
If you're using user_data in your launch template(s), this will only be run when the server has a working network connection. If a server has no working network connection, user_data cannot be retrieved from the instance metadata and cannot be run.
Make sure that the customer tests the application(s) during migration and sign off on them. Ultimately, it's the customers responsibility to determine if an application is working as intended and if all data is correctly transferred.