DEV Community: Matt Bacchi

Using the VSCode Claude Code Extension with Bedrock and Claude Sonnet 4.5

Matt Bacchi — Fri, 02 Jan 2026 03:55:36 +0000

Lots of folks use the Claude IDE, or the Claude Code VSCode extension. Unfortunately, your prompts and completions are used (by default) to train Claude models. [0]

AWS Bedrock, on the other hand, doesn't use your prompts and completions to train any AWS models or give them to 3rd parties. [1]

For these reasons (privacy, data sovereignty) I'm more inclined to use Bedrock as an LLM in my IDE. Today we'll go over how to set up the VSCode Claude Code extension with AWS Bedrock and use the Claude Sonnet 4.5 foundation model.

Overview

In order to use the Claude Code VSCode extension with Bedrock and the Claude Sonnet 4.5 model, we need to perform these tasks:

Setup AWS IAM permissions to allow Bedrock usage
Install and configure Claude Code VSCode extension
Integrate AWS credentials, configure extension
Test our setup works

To do this, we'll use Terraform to create our AWS IAM user and policies (but you could use the AWS console.)

Then we'll integrate this all together and verify it works as expected.

Unfortunate Missing Features and a Bug/Regression with the Claude Code Extension

AWS Bedrock enables generating short lived API tokens via their SDK. [2] Claude Code does support two methods for automatic AWS credential refresh, but Bedrock API tokens is not one of them.

If it did support this feature it would be the best solution from a security perspective. Tokens would expire in 12 hours and when expired and the extension is used it would automatically refresh them.

Instead it only supports AWS SSO (or rather AWS Identity Center) for it's awsAuthRefresh option, or AWS IAM credentials for its awsCredentialsExport refresh methods. [3] This deficiency is a poor decision, or at least an oversight by the Claude Code development team.

Unfortunately, a more egregious issue is that they claim the above awsCredentialsExport refresh method is functional when it is not. Whether it's a regression or bug, or maybe never worked, I couldn't get it working within a couple hours. (Including time spent conversing with the Claude Code VSCode extension using a working AWS Profile and it couldn't suggest a workaround to this problem.) In addition to all these setbacks, using the Claude Code settings file (~/.claude/settings.json) didn't work either so I have to use the VSCode settings file to set all Claude Code extension configuration options.

Since using AWS Identity Center for a personal account is overkill, refreshing Bedrock API tokens in Claude Code is not supported, and the AWS credential export method to automatically refresh my credentials for the Claude Code VSCode extension is not functional, I'll settle for the lowest common denominator and use the AWS Profile in my configuration. I don't love this method because it uses a long lived AWS IAM user credential (access key and secret access key.) But I can't improve security due to the poor state of affairs in the Claude Code VSCode extension.

Create IAM User and Bedrock Permissions

Create an IAM user and attach IAM policy to the user with this Terraform:


resource "aws_iam_user" "bedrock_user" {
  name = "bedrock-user"
}

resource "aws_iam_access_key" "bedrock" {
  user = aws_iam_user.bedrock_user.name
}

data "aws_iam_policy_document" "bedrock" {
  statement {
    effect = "Allow"
    actions = [
      "bedrock:InvokeModel",
      "bedrock:ListFoundationModels",
      "bedrock:ListInferenceProfiles",
      "bedrock:InvokeModelWithResponseStream",
      #   "bedrock:CallWithBearerToken" # required if using Bedrock API token which we're not doing here
    ]
    resources = ["*"]
  }
  statement {
    effect = "Allow"
    actions = [
      "aws-marketplace:ViewSubscriptions",
      "aws-marketplace:Subscribe"
    ]
    resources = ["*"]
    condition {
      test     = "StringEquals"
      variable = "aws:CalledViaLast"
      values   = ["bedrock.amazonaws.com"]
    }
  }
}

resource "aws_iam_user_policy" "bedrock" {
  name   = "bedrock"
  user   = aws_iam_user.bedrock_user.name
  policy = data.aws_iam_policy_document.bedrock.json
}

Setup AWS Profile

I typically use aws-vault to manage my AWS credentials, for enhanced security and to use short lived credentials. But again we're going to have to use the standard AWS method of storing long lived credentials in our ~/.aws/credentials file, and then access that profile from the Claude Code extension.

Above in the IAM user creation section, we only created the IAM user and policy. Now you will need to create an Access Key to use as the credential in your profile. Go to the AWS Console and create an access key for this user, follow the instructions to create it for the CLI use case.

After creating it, copy your Access Key and Secret Access Key somewhere for safe keeping (I use a password manager for this purpose.)

To setup your profile in the ~/.aws/credentials file, use the command:

aws configure --profile YOUR_AWS_PROFILE_NAME

It will prompt you to provide the AWS Access Key and Secret Access Key, which will be added to the ~/.aws/credentials file using the profile name you specified in the command (choose wisely!)

Enable Bedrock Claude Sonnet 4.5 and Validate Access

After creating the above IAM user and policy, and setting up the profile, we'll login to the AWS console. Choose the AWS region that you normally use, but be aware that these Bedrock foundation models aren't available in every single global AWS region. I used region us-west-2 but us-east-1 and us-east-2 are also supported.

Anthropic requires first-time customers to submit use case details before invoking a model once per account or once at the organization's management account. [4] This is a rather antiquated policy in the cloud era, but required nonetheless.

Go to the Bedrock section of the AWS console, then to "Chat/Text Playground" and select Anthropic Claude Sonnet 4.5. You'll be presented with a dialog to fill out and enable the foundation model.

Install and Configure VSCode

Install the VSCode extension:

code --install-extension anthropic.claude-code

Identify your Bedrock Claude Sonnet 4.5 inference profile ARN:

aws bedrock list-inference-profiles  --region us-west-2 --profile YOUR_AWS_PROFILE_NAME --no-cli-pager | jq '.inferenceProfileSummaries | .[] | select(.inferenceProfileId | match("us.anthropic.claude-sonnet-4-5-20250929-v1:0")) | .inferenceProfileArn'

NOTE: This assumes you're in the US, if using another global region use the global Anthropic Claude Sonnet profile name in the above command.

Add the following to your VSCode user settings.json file (usually this is ~/.config/Code/User/settings.json):

{
  "claudeCode.selectedModel": "us.anthropic.claude-sonnet-4-5-20250929-v1:0",
  "claudeCode.environmentVariables": [
    {
      "name": "AWS_PROFILE",
      "value": "YOUR_AWS_PROFILE_NAME"
    },
    {
      "name": "AWS_REGION",
      "value": "YOUR_AWS_REGION_FROM_ABOVE"
    },
    {
      "name": "BEDROCK_MODEL_ID",
      "value": "INFERENCE_PROFILE_ARN_FROM_ABOVE"
    },
    {
      "name": "CLAUDE_CODE_USE_BEDROCK",
      "value": "1"
    }
  ],
  "claudeCode.disableLoginPrompt": true,
}

Did it work?

You'll probably have to restart your VSCode session if it was running. Then, open the Claude window and type a question or request and you should see a successful response like below:

Conclusion

We did it! I'm not super impressed with the missing/broken/overlooked features of the Claude Code VSCode extension related to AWS IAM credentials. But it works fine for the time being and I'll revisit this issue and report back when these issues are resolved and we can all begin using short lived credentials with the extension.

Resources

I borrowed some information from this incredibly detailed blog post by Vasko Kelkocev. [5]

But even though that blog was written in October 2025, it was already out of date by the time I found it. I had to add more IAM permissions to get the extension to work with Bedrock (specifically bedrock:InvokeModelWithResponseStream), and there were some other issues with the configuration I had to play with. Thanks for the great blog Vasko.

Firecracker Virtualization Overview

Matt Bacchi — Thu, 11 Dec 2025 22:52:19 +0000

Firecracker is an open source virtualization technology created by Amazon Web Services (AWS) which underpins their AWS Lambda Functions as a Service (FaaS) serverless product.

Firecracker was open sourced in 2018 [0], making it possible for anyone to use this extremely fast and reliable system for their own projects and use cases.

I've been researching the ecosystem lately and am impressed at the flexibility architected into the Firecracker code which enables it to be used in many ways. We might have expected them to design it such that it only works in their very tightly controlled environment. But the fact that it's not specialized to just the AWS Lambda use case means that it can be leveraged by anyone from AWS scale to a home lab running a single VM.

Let's explore the capabilities of Firecracker and the various methods of using it.

Firecracker High Level Overview

There are many good quick start documents 1 and blogs describing how to install and start a single Firecracker MicroVM instance. Because of these great resources readily available, I won't describe that here.

The basic requirements are:

Firecracker binary
kernel
rootfs
networking configuration (optional)

When you start a firecracker VM by executing the firecracker program, you are running a single VM instance which can be managed through the firecracker process you launched it with. You can send subsequent commands to perform additional management tasks to the firecracker process using the API via a unix socket. When you are done with the VM you can and should stop it "gracefully".

Multiple firecracker processes can be executed at one time, which translates to running multiple VMs. Presumably this is how AWS runs millions of Lambda functions.

Configuration

There are a couple ways to configure your VM:

Sending HTTP requests via the API socket [2]
Using the configuration file [3]

If you use the configuration file to set the VM guest kernel and rootfs, you can still use the socket to send API requests. These configuration methods seem to conflict, but they appear to me to be part of the design of a highly flexible system.

The HTTP API is enhanced by what appears to be an official Go SDK [4] that you can use to manage your Firecracker VM instances. A likely scenario is that AWS uses this Go SDK to provision and control their Lambda functions via this HTTP API provided by the Firecracker process.

Initialization

Firecracker is also flexible in how it can be initialized. Options include:

command line execution
systemd
jailer

The first two options are obviously common to many Linux tools. But the diverse use cases that AWS themselves have for a multi-purpose virtualization technology like Firecracker make it likely that they designed it to be versatile. This adaptable capability extends from the configuration methods to initialization, and likely beyond.

Enhanced Isolation using Jailer

The jailer tool [5] which is specifically designed for firecracker further isolates each firecracker process to its own chroot jail directory. This security mechanism has been used for years in Unix systems and affords a greater sense of security in a multi-tenant environment.

We won't go into detail on this for now, and it isn't necessary for your home lab unless you're running untrusted code (i.e. not written by you.)

Demonstration

To quickly show what we've discussed above, we'll demonstrate with a small bash script that starts a Firecracker micro VM using previously downloaded kernel and rootfs. Here's the script:

#!/usr/bin/env bash

firecracker="/usr/bin/firecracker"

uuid=$(/usr/bin/uuidgen)

nohup $firecracker --api-sock /tmp/$uuid.socket &

echo "created firecracker process running on socket: /tmp/$uuid.socket"

curl --unix-socket /tmp/$uuid.socket -i \
    -X PUT 'http://localhost/boot-source' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
        "kernel_image_path": "/tmp/firecracker-test/hello-vmlinux.bin",
        "boot_args": "console=ttyS0 reboot=k panic=1 pci=off"
    }'

curl --unix-socket /tmp/$uuid.socket -i \
    -X PUT 'http://localhost/drives/rootfs' \
    -H 'Accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
        "drive_id": "rootfs",
        "path_on_host": "/tmp/firecracker-test/hello-rootfs.ext4",
        "is_root_device": true,
        "is_read_only": false
    }'

curl --unix-socket /tmp/$uuid.socket -i \
    -X PUT 'http://localhost/actions' \
    -H  'Accept: application/json' \
    -H  'Content-Type: application/json' \
    -d '{
        "action_type": "InstanceStart"
    }'

echo "Started VM instance."

Running it we get the following output:

user@temphost:~/data/firecracker-test$ ./startvm.sh 
created firecracker process running on socket: /tmp/b214c708-e506-455b-a98d-647939a0ef0d.socket
nohup: appending output to 'nohup.out'
HTTP/1.1 204 
Server: Firecracker API
Connection: keep-alive

HTTP/1.1 204 
Server: Firecracker API
Connection: keep-alive

HTTP/1.1 204 
Server: Firecracker API
Connection: keep-alive

Started VM instance.
user@temphost:~/data/firecracker-test$ tail nohup.out 
 [ ok ]
 * Mounting persistent storage (pstore) filesystem ...
 [ ok ]
Starting default runlevel
[    1.088111] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2879b124109, max_idle_ns: 440795245440 ns

Welcome to Alpine Linux 3.8
Kernel 4.14.55-84.37.amzn2.x86_64 on an x86_64 (ttyS0)

localhost login:

You can see that we have a login prompt in the nohup.out file that indicates the VM is running and ready to accept logins. We didn't setup any SSH keys in this attempt, nor networking, so we're SOL on logging in. But that's OK because we were just trying to show how straightforward this can be.

Notice we created a random UUID to use as the firecracker unix socket, which is how you can communicate with it via the HTTP API. After starting the firecracker process, we then make curl requests to configure the kernel, rootfs, and then initiate the InstanceStart action.

Granted, this is a very rudimentary example. But it shows how easily firecracker can be used if you understand the fundamentals.

Firecracker Ecosystem Discrepancies Compared to Lambda

In multiple AWS re:Invent talks, Lambda architecture has been detailed by AWS employees at a high level. One example is in Julian Wood's 2022 session A closer look at AWS Lambda (SVS404-R). At 10:36 in the video, he references the Lambda data plane, which consists of a host of interrelated services for synchronous vs. asynchronous Lambda invocations. These services are:

Synchronous

Frontend Invoke Service
Counting Service
Assignment Service
Worker Hosts
Placement Service

Asynchronous

Poller Fleet
Queue Manager
State Manager
Stream Tracker
Leasing Service

The Cold Truth

These administrative services used to run millions or billions of AWS Lambda invocations are obviously not available for Firecracker. It makes sense that AWS would release their Firecracker product as open source, but they're not likely to ever provide the facade that props Firecracker up in their environment, enabling AWS Lambda to be highly performant and reliable. But this is something I'm interested in. I think some of these features that streamline the functionality of Firecracker in the Lambda use case could be built to make it easier to manage for those of us who think this tool is exciting and wish it came with batteries included.

Conclusion

Over the next few weeks (and months) I'll be playing with this some more. I'll likely be posting as I do this to give a different perspective on firecracker than the standard quick start blogs out there already.

Hope you got something out of this today!

Installing Nvidia Datacenter GPU Manager on Amazon Linux 2023

Matt Bacchi — Sat, 01 Feb 2025 15:51:13 +0000

A short post discussing the installation of Nvidia Datacenter GPU Manager on Amazon Linux 2023. I recently had to figure this out using sketchy documentation, so I'm hoping this helps some folks out there doing similar head scratching.

NOTE: This blog post does not include all steps to install Nvidia drivers. I assume you already have driver package(s) required for your application installed.

What is the Datacenter GPU Manager

Nvidia Datacenter GPU Manager is a suite of tools used for managing and monitoring Nvidia GPUs in the data center.

In our environment we use Nvidia Datacenter GPU Manager as a prerequisite for the DCGM Prometheus metrics exporter used to send metrics to Datadog, named dcgm-exporter. I won't go into detail on installing dcgm-exporter, the instructions in the Github README are fairly straightforward to follow.

Amazon Linux 2 End Of Life

The Amazon Linux 2 (AL2) EOL date is 6/30/2025 (June 30, 2025). The replacement is Amazon Linux 2023 (AL2023) [3].

Installing DCGM on AL2023

Installing Nvidia Datacenter GPU Manager on AL2 was documented, but applying that documentation to AL2023 wasn't clear what value to use for the <distro> [4].

In order to understand what distribution Amazon Linux 2023 most closely relates to, you have to look in the Amazon Linux 2023 User Guide. It describes that AL2023 is "sourced from multiple versions of Fedora...including CentOS 9."

That means when following the Nvidia installation instructions, they suggest enabling the repository with this command:

sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/<distro>/x86_64/cuda-<distro>.repo

Because AL2023 is derived from Fedora/CentOS 9, the <distro> parameter is going to be rhel9. So our command to install the repository instead becomes:

sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo

Now you can run the install command for the datacenter-gpu-manager package:

sudo dnf install -y datacenter-gpu-manager

Conclusion

The documentation was not clear which distribution should be used for AL2023. Replace <distro> with rhel9 in the repo installation command and we're now able to install the datacenter-gpu-manager package.

Cover photo by Christian Wiediger on Unsplash

Switching to the Terraform S3 Backend with Native State File Locks

Matt Bacchi — Wed, 08 Jan 2025 15:56:03 +0000

Terraform is a flexible, cloud agnostic infrastructure as code (IaC) tool. As it constructs infrastructure resources, it builds a ledger used to track resources that have successfully been created as well as additional metadata (such as id.) Terraform stores this state in a binary formatted file with the extension .tfstate.

What is the Terraform S3 Backend

The Terraform state file described above by default is stored in the same directory as the Terraform infrastructure definition files you wrote. But with this state on your local computer it is vulnerable to being lost or overwritten, and it cannot be shared with or managed by other team members. Using a distributed storage mechanism to store this state file is straightforward with Terraform, and they provide many backend options. For AWS users, the Terraform S3 Backend allows storing this state file in AWS S3.

What is State File Locking

So now we know this state file is stored in distributed object storage (AWS S3,) and more than one user can manage resources within it. But to safely manage this state file, we require a locking mechanism (often called a mutex in computing) that disallows multiple users from attempting to write to it at once. We would have a mess if we allowed more than one user to write to it at the same time, potentially losing the resources that were created and intended to be stored in this state file.

Terraform state locking capability has been available for the S3 backend for quite some time. But unfortunately it has required an additional DynamoDB table to be created that tracked the state file locking status.

Until now.

This DynamoDB table is an extra resource that seemed tangential to the Terraform state backend process and complicated the process of configuring your backend. That requirement has been rendered obsolete with a recent feature that was added to AWS S3, conditional writes.

AWS S3 Conditional Writes

In August, AWS announced the addition of the S3 Conditional Writes feature. This feature of AWS S3 compels S3 clients to check for the existence of an object before writing it, and if it already exists to fail. If the file exists the S3 client returns a 412 Precondition Failed error response.

S3 Conditional Write Support Added in Terraform v1.10

Support for S3 Conditional Writes was added to Terraform release v1.10. (If you want to see some great background and architecture detail from the developer Bruno Schaatsbergen about the implementation look here.)

Thankfully this is completely transparent to the Terraform user (unless it returns an error attempting to lock the state file.)

Configuring the S3 Backend to Use Native State File Locking

The Terraform documentation describes the new configuration parameter use_lockfile to enable S3 state locking. It also currently describes the old DynamoDB method as still available. (It's common for software to support both an old and new related feature for some time until all users can migrate to the new methodology.)

This means you can actually use both locking mechanisms at the same time. But this is both unnecessary overkill, and could lead to confusion and problems. I would recommend that you replace your old DynamoDB locking configuration with S3 state locking immediately. It will be cheaper (without having to pay for an extra DynamoDB table or reads/writes to that table,) and less error prone.

Here's how to change your Terraform backend configuration.

Terraform Configuration Specifics

The old DynamoDB method used a configuration parameter named dynamodb_table.

The new S3 state locking method uses a configuration parameter named use_lockfile.

Both are covered in the current Terraform documentation.

Version Constraints

We also recommend that when you switch to the S3 native state locking method, you set the Terraform configuration parameter required_version to the minimum version that supports S3 native state file locking. If you have users with an earlier version of Terraform, they won't be able to use this feature and will see errors if the use_lockfile parameter is enabled. Setting the required_version to v1.10 at a minumum makes your configuration more resilient and doesn't let someone attempt to create or update resources using an older Terraform version. Think of it as a prerequisite.

This Terraform version constraints configuration is documented here, and looks like this:

  required_version = "~> 1.10"

Sample Configuration

With all this background information about the configuration parameters, here's a sample Terraform configuration with both the old and new parameters present:

provider "aws" {
  region = "us-west-2"
}

terraform {
  backend "s3" {
    encrypt        = true
    bucket         = "tfstate-lock-test-0bhfxn8x1"
    region         = "us-west-2"
    key            = "example/terraform-state-lock-test.tfstate"
    dynamodb_table = "tfstate-lock-test"
    use_lockfile   = true
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.82.2"
    }

  # This sets the version constraint to a minimum of 1.10 for native state file locking support
  required_version = "~> 1.10"
}

In order to switch away from using the old DynamoDB locking method, remove that dynamodb_table configuration parameter.

State File Locking in Action

We now know how to configure Terraform S3 native state file locking, but how does it perform and what will we see if you cannot get the mutex to lock the file?

I've tested both methods and will show you the output from each when state file locking fails.

Error from DynamoDB State File Locking

The old DyamoDB state file locking method would return an error such as the below:

$ terraform apply plan.out 
Acquiring state lock. This may take a few moments...
╷
│ Error: Error acquiring the state lock
│ 
│ Error message: operation error DynamoDB: PutItem, https response error StatusCode: 400, RequestID: CP9U4IRC04OBONKIQHM4LUOLLJVV4KQNSO5AEMVJF66Q9ASUAAJG, ConditionalCheckFailedException: The conditional request failed
│ Lock Info:
│   ID:        39f64263-4ad8-a563-faf7-f28f8a042a00
│   Path:      tfstate-lock-test-0bhfxn8x1/example/terraform-state-lock-test.tfstate
│   Operation: OperationTypeApply
│   Who:       user@hostname
│   Version:   1.10.3
│   Created:   2025-01-08 03:38:13.121564614 +0000 UTC
│   Info:      
│ 
│ 
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.

Error from S3 Backend Native State File Locking

The new S3 backend native state file locking method will return an error that looks like this:

$ terraform apply plan2.out
╷
│ Error: Error acquiring the state lock
│ 
│ Error message: operation error S3: PutObject, https response error StatusCode: 412, RequestID: N0BGGAFN8V1N2WCQ, HostID: 4G32Xus/86u8CehvNbvzv8NoqiyTvsBWGYBXYK6E8Vn0E4+wom+6Jm6WFVUFSaCE7C1TBP5Vauo=, api error PreconditionFailed: At least one of the pre-conditions you specified did not hold
│ Lock Info:
│   ID:        837482e8-441e-9ff6-d30b-333ee83d8fc4
│   Path:      tfstate-lock-test-0bhfxn8x1/example/terraform-state-lock-test.tfstate
│   Operation: OperationTypeApply
│   Who:       user@hostname
│   Version:   1.10.3
│   Created:   2025-01-08 03:43:11.691479913 +0000 UTC
│   Info:      
│ 
│ 
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.

Dealing with Stale State File Locks

NOTE: After publishing this blog, I was asked whether the terraform force-unlock command still worked. I tested this and can say it does perform as expected with the old S3 DynamoDB state file locking mechanism. Here's an example session showing this:

$ terraform apply plan2.out
╷
│ Error: Error acquiring the state lock
│ 
│ Error message: operation error S3: PutObject, https response error StatusCode: 412, RequestID: NGQM2VGSTSDWPCZF, HostID: dzPZArnTy31oVeuVLI8Dm61HXnuL6M3R2tlWFe2suztP0zkh4Bwv/eJFBLqVfitAI40I5BvIeds=, api error PreconditionFailed: At least one of the pre-conditions you specified did not hold
│ Lock Info:
│   ID:        bde40e3b-2bfb-f577-fea5-44923c9d5275
│   Path:      tfstate-lock-test-0bhfxn8x1/example/terraform-state-lock-test.tfstate
│   Operation: OperationTypeApply
│   Who:       user@hostname
│   Version:   1.10.3
│   Created:   2025-01-08 16:55:35.808464751 +0000 UTC
│   Info:      
│ 
│ 
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵

You can remove the lock, but only do this if you know the lock is stale. To do this, first note the lock ID above, then run the force-unlock command:

$ terraform force-unlock bde40e3b-2bfb-f577-fea5-44923c9d5275
Do you really want to force-unlock?
  Terraform will remove the lock on the remote state.
  This will allow local Terraform commands to modify this state, even though it
  may still be in use. Only 'yes' will be accepted to confirm.

  Enter a value: yes

Terraform state has been successfully unlocked!

The state has been unlocked, and Terraform commands should now be able to
obtain a new lock on the remote state.

Delete Your Old DynamoDB Tables

Now that you've switched from using the old Terraform DynamoDB locking to the new S3 native state file locking, you can remove the old DynamoDB table used to track these locks!

Yay, one less resource to manage and be charged by AWS for.

Summary

Hopefully you see the advantage of using the new Terraform S3 backend native state file locking mechanism, and how to configure it for your environment.

Happy Terraforming!

A Survey of Serverless Sustainability Trends

Matt Bacchi — Mon, 30 Dec 2024 02:13:44 +0000

As we bring 2024 to a close, and after an invigorating week at AWS re:Invent, many will be writing their year in review summaries. I've decided to dedicate those column inches to the state of serverless sustainability today. The observant among us are quite aware of how the artificial intelligence (AI) craze has wormed it's way into every product, industry and conversation over the past year. It's been making headlines as shuttered power plants like Three Mile Island are reopened. And it's on a collision course with the world's climate change goals. (AWS also announced in October they signed agreements for 3 small modular nuclear reactor projects for their own data centers.)

Running servers in data centers is an energy intensive process. Every search for a restaurant or game score requires a response from a server. Running a busy webserver requires the same power as one that gets 5 hits per day. Serverless technologies like AWS Lambda, S3, DynamoDB can be used to reduce that constant power usage when there is no demand.

Today we'll discuss the current state of affairs with serverless sustainability, and perhaps a little about sustainable computing in general.

What is Sustainability

In the context of technology and computing in particular, sustainability equates primarily to power usage and carbon emissions. The computing industry has been on an upward trend in energy use since, well, the beginning of the computer era. During that time, increases in computational capabilities (faster processors) have driven higher demands for electricity. The next (current?) era of computing is going to require rethinking these needs by reducing the power requirements of computers and data centers, recycling hardware, and becoming better climate citizens. (AWS discusses some of their sustainability improvements related to the circular economy in this re:Invent talk "Advancing sustainable AWS infrastructure to power AI solutions" here.)

AWS defines sustainability as one of the 6 pillars of their Serverless Applications Lens for the AWS Well-Architected Framework. Their Well-Architected Framework documentation covering Sustainability indicates their focus on environmental sustainability in this pillar. It's important that they make that distinction, and even more telling that they are making a large commitment to improving energy efficiency and carbon emissions in their own operations. This document (and others they've released [PDF]) lay out their goals as well as identify how AWS cloud customers can optimize their operations to become more sustainable.

Sustainable Workloads

As is common when working with AWS, they assert customers approach cloud sustainability by using the shared responsibility model. This is akin to their approach for security in the cloud, so it makes sense that they want users to take our part seriously.

Region Selection

One way to make your application more sustainable is to use the lowest carbon intense AWS region available, taking into account other business requirements such as latency. If you are a small company with only customers in one country or region, you may not be able to take advantage, but if business requirements don't disallow using a more remote location you can use this technique. Obviously some AWS regions are in locations that have a higher percentage of renewable energy entering it's power grid. Deciding to use one of these regions should be based on latency of course too, but carbon intensity must be part of the decision.

How, then, do we choose a region for sustainability? A good reference is this AWS Architecture Blog: How to select a Region for your workload based on sustainability goals. They suggest using the site electricitymaps.com to identify carbon intensity and renewal energy percentage for each regional electricity grid.

On the electicitymaps 24 hour climate impact map, hover over the PJM region (which covers Virginia, home to the us-east-1 AWS region) we see the carbon intensity (as of 12/21/2024) is 409g CO²/kWh, and the renewable energy mix is 8%. If we now look at the California power grid (home to the AWS us-west-1 region), we see 139g CO²/kWh, and the renewable energy mix is 67%.

Over-provisioned Capacity and Time Shifting

Earlier this year I ran across a podcast from the Green Software Foundation called Environment Variables. In one episode the guest Kate Goldenring talked about the sustainability merits of serverless. She pointed out a statistic from the Sysdig 2023 Cloud Native Security and Usage Report that showed 69% of requested CPU resources go unused. That means as an industry we're over provisioning by more than two thirds (2/3). The goal should be higher hardware utilization according to Kate.

This proves what we've been saying about how running applications on servers (such as EC2) or containers is less efficient than serverless. Provisioning for peak capacity is what makes these "always on" deployments wasteful because traffic is rarely at it's expected peak usage. Serverless technologies that utilize multi-tenancy are intentionally more sustainable than traditional deployment methods because they enable higher hardware utilization.

It's idealistic to assume we can perfectly utilize hardware via multi-tenancy, but one approach Meta has begun using in their internal private cloud functions product called XFaaS (similar to AWS Lambda) is a technique called "time shifting". Instead of running all workloads immediately, they will delay some operations that are not time sensitive. This increases the average utilization of their hardware due to scheduling these delay tolerant functions during off-peak times, ultimately lowering their peaks and raising their troughs. That equates to a higher average utilization.

This introduces an alternative approach to managing sustainability, seldom discussed in the serverless realm as far as I've seen. In many event driven systems events don't require immediate processing, and are delay tolerant by design. This characteristic lends itself well to having the processing delayed until utilization in your system is below a certain threshold (i.e. off hours.) Of course this is only possible for data that isn't expected to be immediately consistent. This type of processing emerged in the mainframe computing era, and was called "batch processing". It's still in use today, did you ever wonder why your bank statement can take up to 24 hours to reflect purchases? They're processing payments either overnight or in some batch processing methodology.

Time Shifting on Kubernetes

Although I don't consider Kubernetes to be serverless by definition, I happened across a Kubernetes admission controller recently that accepts delay tolerant workloads. This paradigm is atypical for Kubernetes (which usually processes a request as soon as it's received,) but the pattern hints at the potential future of all workload processing systems.

You may have a request sent to your backend system that doesn't require immediate processing, but rather is capable of running whenever is most efficient to do so. The research paper describing the Cucumber and a (2 year old but unmodified) reference implementation are available. While this is more geared towards running workloads when excess solar power is being generated, the concept could be extended to many types of situations such as how heavily loaded your servers are.

Summary

You've just read a few of the trends that have been developing in the sustainable serverless computing arena. Hope this has provided some new information for you to go delve into on your own. Thanks and have a great 2025!

Cover photo by Samuel Faber from Pixabay

Limitations of AWS EC2 Image Builder Lifecycle Policies

Matt Bacchi — Tue, 13 Aug 2024 13:13:14 +0000

EC2 Image Builder Lifecycle Policies is a fairly new AWS feature that enables automatic cleanup of old EC2 Image Builder pipeline artifacts. If you haven't used EC2 Image Builder, it's quite handy for creating customized AWS Machine Images (AMIs) for your EC2 deployments.

This could be a really valuable tool that simplifies the AMI build process. But there are a number of artifacts left behind after your AMI is built that potentially cost money and require manual effort to clean up.

Enter EC2 Image Builder Lifecycle Policies. This feature is advertised as a way to automatically tidy these artifacts. Unfortunately it's almost unusable if you build a large number of AMIs constructed from numerous recipes.

Below I'll describe a high frequency use case that illustrates the limitations of EC2 Image Builder Lifecycle Policies. You can make your own conclusions about the utility of the service as it stands today.

Standard use case

The process of building AWS EC2 Amazon Machine Images (AMIs) can be rather involved. You start with a base image, add required packages and user data, then build, test, and finally deploy your custom AMI. The deployment step isn't even straightforward, as it potentially requires disk volumes, network interfaces, and additional resources which necessitate experimentation. Even after you've done all this, managing the state of these AMIs and their related resources still isn't complete. These resources all need to be cleaned up. And because AMIs tend to be created frequently to meet security objectives, the entire process can occur at a high cadence.

Why so many AMIs?

We all know how frequently CVEs (Common Vulnerabilities and Exposures) can be released, and how quickly management expects them to be mitigated. Lower severity security updates come out at an even more unrelenting pace. For production infrastructure at enterprise scale, updates need to be routinely applied and be extremely reliable. This process is simplified by automating it all with an EC2 Image Builder pipeline executed on a regular schedule.

But after these AMIs and related resources have been superseded by newer ones they need to be removed. Until now there was no method provided by AWS to perform this action. You needed to develop your own tooling for this purpose.

EC2 Image Builder Lifecycle Policies

Lifecycle Policies adds a necessary (heretofore missing) feature to EC2 Image Builder. Having AWS remove your unused resources is a great idea. But the execution leaves something to be desired. For the sake of illustration, let's also define what "related resources" means here.

Every AMI built via EC2 Image Builder requires an image recipe and image pipeline be created. An image recipe consists of AWS managed components or custom (user generated) components to perform an AMI build. The pipeline runs your recipes which generate an output image upon success which references the output AMI, infrastructure configuration, distribution settings, and a CloudWatch log stream (which includes pipeline output for debugging purposes.)

When you modify the image recipe, a new recipe version is created and the previous versions become obsolete. The now unused recipe and it's related resources (such as the output image as well as the AMI) remains active in AWS indefinitely and won't be removed until you perform that action. (I've highlighted this specific point for a reason that will become obvious later.)

If you thought you merely needed to delete the AMI when you were done with it, you would leave all these related Image Builder resources in your account. (Granted they don't cost a lot, but it makes sense to remove unused resources, if only to avoid hitting quota limits at some point in the future.)

While EC2 Image Builder Lifecycle Policies makes this last cleanup bit somewhat more straightforward, it cannot be said it's "a one size fits all" solution.

Limitations of the current Lifecycle Policies service

To create a lifecycle policy for AMI image resources, the documentation says you must "apply lifecycle rules to image resources based on the recipe that created them, select up to 50 recipe versions for the policy."

This is unwieldy because you have to add new recipe versions explicitly each time you create a new image recipe. If you use Infrastructure as Code (IaC) tools to do this (such as Terraform, CloudFormation or CDK), it becomes slightly less manual, but still requires manual effort (or at least in depth awareness of the process.) But even if you add these recipes to your Lifecycle Policy manually, there's a limit of 50 recipe versions in the policy. This isn't as irritating if you only have a dozen recipes in total, the scope of the problem is intensified with scale.

Also, I'm referring to lifecycle policies for AMI image resources, I don't currently use EC2 Image Builder to create container image resources.

The limitations as I see them are:

The maximum number of recipes allowed in the LifecyclePolicyResourceSelection API documentation which can be used as selection criteria is 50. This 50 recipe limit is quite an arbitrary number, and when a user has a large number of image recipes it is both an extremely small limit, and would become a burden to add all recipes to this rule scope parameter.

I've verified this is not adjustable in the AWS Service Quotas self service quota increase dashboard in the AWS Console (or the Service Quotas API).
In the EC2 Image Builder Lifecycle Policies AWS Console, when selecting the recipes to apply the Rule Scope, the drop down list only displays a few current recipes. It shows a spinner and indicates it is "Loading Recipes" (see screenshot) but the additional recipes never show up.

Conclusion

While EC2 Image Builder Lifecycle Policies has the potential to be a great service, it is currently hampered by the user experience (UX.) Adding functionality to make adding recipes to policies easier would be a major improvement for early adopters and power users.

Event Driven Processing of ip-ranges.json

Matt Bacchi — Thu, 01 Feb 2024 22:32:10 +0000

Imagine you have a security group that needs to allow all IP addresses of AWS EC2 instances. Or imagine you have to allow IP addresses of Github Actions runners so that only your CI workers connect to your VPC. Both of those IP address ranges change regularly, and need to be updated (usually by hand.)

If we want to automate these security group updates, how could you figure out when these IP address ranges have changed? AWS has an SNS notification sent every time their ip-ranges.json list changes. The SNS notification can be used to initiate an automated procedure to update our security group.

What we're describing is an event driven architecture. In event driven architectures, an event producer causes an event to be created. A downstream event consumer handles the event and may trigger further events.

In this 2 part blog post series, we'll cover event driven architectures. In the first part of the example used to illustrate the mechanics, we'll build the IP address range processing component. This piece of the puzzle processes the AWS ip-ranges.json file and inserts it into a DynamoDB table. The second blog in this series will insert the IP ranges into AWS security groups.

After the next blog post, you'll be able to manage IP address ranges in your environment using Event Driven Architecture.

What is Event Driven Architecture?

The Wikipedia page for event driven architecture (or EDA) describes it as "a software architecture paradigm concerning the production and detection of events". This is in contrast to software which is focused on its own state, and doesn't concern itself with external state changes. Event driven architectures are often made up of loosely coupled components that act independantly on events that they're concerned with.

Why use EDA for data processing?

Data processing is often well suited to the EDA model because it's rarely necessary to syncronously perform a data processing task. If a user purchases something in a shopping cart, certainly the interatction with the bank or credit card needs to be syncronous and immediate. But if the website catalogs all purchases in a "top ten products" list from all user purchases, that processing can be done asyncrynously at a later, and possibly less busy time of the day. It's common for data engineering teams to perform extract, transform, load (or ETL) jobs that process data off hours, to avoid contention for compute resources. This can all be done with an event driven model, where the list of purchased items in our above example would be sent to an event bus (such as AWS EventBridge or Apache Kafka.) Consumers of that type of event would then act on the event depending on their function.

SNS notification for ip-ranges.json

As mentioned above, anyone can subscribe to SNS notifications for the ip-ranges.json list. When a change occurs to that list the SNS notification is sent out. It acts as the event producer in this case, causing anyone who is subscribed to the notification list to receive the event.

In our example today, we'll setup an AWS Lambda function to process these events stemming from the SNS notification. Our function will be the event consumer in this scenario.

Data organization of the ip-ranges.json file

The ip-ranges.json file syntax consists of a creation date (to indicate last update time) and a list of IPv4 and IPv6 address ranges. These look like the following sample:

    {
      "ip_prefix": "52.4.0.0/14",
      "region": "us-east-1",
      "service": "EC2",
      "network_border_group": "us-east-1"
    },

We want to maintain all of this information, and in fact enhance it with the date stamp (or synctoken) which is included in the ip-ranges.json file as mentioned above.

DynamoDB Single Table Design

We'll be inserting all of these IP address ranges into a DynamoDB table for future access and processing (triggered via events.) DynamoDB is great for serverless applications because it allows a large number of stateless connections via HTTP. Using a single DynamoDB table, we'll store all of this data in a way that can be easily queried by creating composite keys that pre-join the data fields and make lookups extremely fast. This concept is discussed quite extensively, but Alex Debrie (an AWS Data Hero) has a blog post that covers these ideas quite well.

Since we want to enable queries that return IP prefixes based on AWS region and service. In order to do that, we'll format the data with both a primary key (PK) and sort key (SK), as well as a synctoken that is completely separate from the IP prefixes. This synctoken will enable us to remove all IP prefixes that have the synctoken because we'll be creating a global secondary index using the synctoken to allow fast queries of the items containing a particular synctoken. We can add non-normalized data to the same DynamoDB table because it isn't an SQL database, it's more like a key-value or wide-column store database, generally referred to as NoSQL.

Here is what our primary and secondary key layout will look like:

Lambda Function Overview

We'll use a Lambda function that gets triggered by an SNS notification whenever the IP ranges JSON changes. This SNS subscription, Lambda Function, DynamoDB table and all the AWS infrastructure is configured using AWS Cloud Development Kit (CDK.) The Lambda Function itself does one thing. It downloads the JSON, cycles through every item, and adds it to the DynamoDB table.

AWS CDK configuration

The configuration used to create the Lambda Function and other infrastructure is in the Github repository here. Instructions on how to deploy the infrastructure are also in that repository.

Once deployed the Lambda Function will do nothing until the next time the SNS notification invokes it when the JSON file has changed. Below you can see a screenshot of recent invocations of the function in my environment.

Wrapping Up

In this first part of the blog series, you saw how we used Event Driven Architecture to respond to an event and perform some data processing. In the next section we'll handle using these IP address ranges to update security groups allowing traffic for certain AWS services.

Thanks for reading and have a great day!

Resources

AWS blog post on "How to Automatically Update Your Security Groups for Amazon CloudFront and AWS WAF by Using AWS Lambda"

Cover photo by Dushyant Kumar on Unsplash

S3 Bucket Takeover Neutralization

Matt Bacchi — Tue, 30 Jan 2024 14:07:57 +0000

There's been a recent uptick in the number of S3 buckets that have become "hijacked" or "taken over".

How can an attacker take over your S3 bucket? What can you do to prevent this on your buckets?

We'll answer these questions and provide details on how you can neutralize the threat of S3 bucket takeovers.

What is Hijacking?

How can your S3 bucket get hijacked?

It should be noted that the term "hijacked" is a misnomer. S3 buckets which have ended up in this state were managed poorly by the owner. The original owner had to choose to give up ownership of the S3 bucket, prior to it being claimed by a bad actor. If you have an active AWS S3 bucket in your account, this cannot occur unless you relinquish control. These are categorized as software supply chain attacks, where the attacker hijacks open source software or software updates, eventually compromising target computers.

AWS uses a global namespace for S3 buckets. If a bucket name "mattbacchi" is available and I requisition it, no one else in the world can use that bucket name. A global namespace has some downsides, namely that everyone competes for the same bucket names, and a single bucket name cannot be owned by more than one AWS account.

This surge in exploit/hijack/takeover incidents ultimately has this global namespace to blame. If you owned an S3 bucket but then deleted it, someone else could subsequently recreate it in their account (after a short period of time.)

If the deleted bucket was public and contained trusted files (such as npm packages), someone with sinister intentions could recreate the bucket and masquerade as you. If they named their tainted files the same as your trusted files, users might download the malicious content assuming it was your original content. This is how the above poisoned "bignum" npm package exploit worked. The S3 bucket owned by someone the "bignum" software developer trusted was abandoned, subsequently an attacker recreated it with malicious content.

These attacks cannot occur if S3 buckets are managed with better data lifecycle hygiene.

Preventing S3 Bucket Takeover

In order to neutralize this threat, I recommend never deleting your S3 buckets.

As described above, the only S3 buckets vulnerable to this attack vector are those that have been deleted and are no longer active. The logical defense is to never relinquish your S3 buckets, instead you maintain the bucket and name forever.

Won't this get expensive, you may ask? AWS bills customers for S3 based on storage. If you aren't storing any data in the bucket, it will never generate a charge. In this way we retain an S3 bucket name so that it doesn't fall into the wrong hands.

Of course, this may not be necessary for all S3 buckets you own. I recommend using this technique for any buckets that are public and have been used in production software applications. The "bignum" npm package bucket above is a prime example, retaining that bucket would have prevented the attack.

Decommissioning Process

Developing an official process to manage this in your organization is important.

If you intend to retain your S3 buckets in order to prevent takeover, you must first develop documentation and communicate this policy. Also consider how you will implement identifiers or signposts indicating the bucket must be retained to administrators who may not be familiar with the process. I've personally used documentation, an AWS tagging scheme, and a couple other methods to prevent the bucket from being accidentally relinquished.

One method of making this obvious to infrastructure engineers (i.e. devops or platform engineers) is to create a zero byte file in the bucket that identifies it as persistent. I have suggested the file be named DO_NOT_DELETE_S3_BUCKET_PERSISTENT as a fairly large signpost for people to prevent them from deleting it. A sample command to do this via the AWS CLI is something like:

touch DO_NOT_DELETE_S3_BUCKET_PERSISTENT; aws s3 cp DO_NOT_DELETE_S3_BUCKET_PERSISTENT s3://BUCKETNAME/

Another method is to create an AWS tag on the bucket such as persistent=true.

Finally, if you want to use a more durable mechanism, add an S3 bucket policy that denies the action s3:DeleteBucket as well as s3:PutObject. This will prevent removing the bucket unless the user is an administrator, at which point they can remove the s3:DeleteBucket permission from the bucket policy. But they must manually perform this update, and hopefully that extra step makes them realize it shouldn't be deleted.

Conclusion

Today we've discussed the cause of S3 bucket hijacking and some methods of preventing it. Hopefully this motivates you to perform some analysis of your S3 buckets and apply these suggestions in your environment.

Thanks for reading and have a great day!

Bundling Go Lambda Functions with the AWS CDK

Matt Bacchi — Fri, 22 Dec 2023 01:38:03 +0000

Recently the Lambda Go runtime has changed from using the Go 1.x managed runtime to using the provided runtimes which have been historically used for custom runtimes (i.e. Rust.) The former go1.x runtime is being deprecated on January 8, 2024 (quite soon) and the new runtimes provided.al2023 or provided.al2 are expected to be used.

With the introduction of these new runtimes, all of our Go binaries must now be called bootstrap and be located at the root of the zip file used to deploy the function.

If we have two functions built by separate function code, in the go1.x runtime the naming convention would allow us to name them differently (i.e. create vs. list). Now we must name both function handlers bootstrap, potentially causing confusion during the bundling process.

In today's blog post, we'll describe how to bundle two different Go functions with the same name in a single CDK stack.

AWS CDK Deployment Using Typescript

I'll be using Typescript as the language for the AWS CDK deployment. It might be strange to mix languages for CDK deployment (Typescript) and the Lambda function (Go), but CDK uses Typescript as its main language and most examples are written in that. The reason for using Go in this example is because many deployments need to use compiled languages for task duration and performance (not necessarily to improve cold start time.)

Source Organization

As described earlier, in the past we could have kept the names of the Go function source files in the same directory and compiled them separately to different file names. Now that the handler file name has to be the same (bootstrap), they can't be output to the same directory any longer.

The layout of the files now looks like the following, with each function in its own subdirectory in the lib/functions directory:

Bundling Go Functions

Now that we have the function code in separate directories, we can bundle them using the Alpha CDK construct for a Go Function, shown below.

export class BasicGoLambdaStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const table = new TableV2(this, 'Table', {
        partitionKey: { name: 'id', type: AttributeType.STRING },
        removalPolicy: RemovalPolicy.DESTROY,
        });

    const createFunction = new GoFunction(this, 'CreateFunction', {
      entry: 'lib/functions/create',
      environment: {
        "DYNAMODB_TABLE": table.tableName
      },
      logRetention: RetentionDays.ONE_WEEK,
    });

    const listFunction = new GoFunction(this, 'ListFunction', {
      entry: 'lib/functions/list',
      environment: {
        "DYNAMODB_TABLE": table.tableName
      },
      logRetention: RetentionDays.ONE_WEEK,
    });
...

The above configuration is obviously fairly straightforward. But before I found the aws-lambda-go-alpha module, I attempted (mostly unsuccessfully) to use the aws-lambda-function Function construct which required using the Code.fromAsset method with the ILocalBundling interface. This was extremely uncooperative, I found multiple blogs and suggestions on how to configure this, and in the end I did get it working, but the solution I came up with was ugly and actually packaged the Go source file in the zip file with the bootstrap handler. Not cool.

It's great that the CDK now supports Go Lambda functions as a full blown construct. While NodeJS functions have been supported for much longer it makes sense to have Go supported the same way, because the aggravation I endured packaging these Go handlers with the standard Lambda Function construct was a serious pain in the ass.

Verifying the Output

Now that we have these defined, we can package the artifacts using cdk synth and verify that the bootstrap binary was created for both functions.

We can see here that the first messages output from the cdk synth command are the two functions being bundled:

$ cdk synth
Bundling asset BasicGoLambdaStack/CreateFunction/Code/Stage...
Bundling asset BasicGoLambdaStack/ListFunction/Code/Stage...
Resources:
...

And when we look in the cdk.out directory which is where our intermediate CDK files are placed during packaging, we see that the two Lambda Function output directories itemized in the stack asset JSON file (called BasicGoLambdaStack.assets.json) have binaries in them (ignore the third .js asset, that's for the cloudwatch logs):

$ tree cdk.out/
cdk.out/
├── asset.4e26bf2d0a26f2097fb2b261f22bb51e3f6b4b52635777b1e54edbd8e2d58c35
│   └── index.js
├── asset.b0bae29696f98febe5e6a655c4f61466ad71ebfa0071b46a15e917a1d307333c
│   └── bootstrap
├── asset.f153e459ffb70033083e7507aaa06c00f4b13a792fad71433c11b5f050078e74
│   └── bootstrap
├── BasicGoLambdaStack.assets.json
├── BasicGoLambdaStack.template.json
├── cdk.out
├── manifest.json
└── tree.json

4 directories, 8 files

After running cdk synth to verify things, we can deploy with cdk deploy.

Success

Now we can execute the functions and see them working as expected:

$ curl -X POST -H "Content-Type: application/json" https://someurl.execute-api.us-west-2.amazonaws.com/todos --data '{ "title": "stuff", "details": "really some stuff"}'
{"id":"80cf566a-937b-4aef-98ef-48b84ee75c01","title":"stuff","details":"really some stuff"}

$ curl -X GET https://someurl.execute-api.us-west-2.amazonaws.com/todos
[{"id":"80cf566a-937b-4aef-98ef-48b84ee75c01","title":"stuff","details":"really some stuff"}]

Summary

The new requirements for using the provided runtimes for Go Lambda Functions poses some minor challenges (especially if you have to migrate already running Go 1.x Lambda Functions) but thankfully the aws-lambda-go-alpha module makes it much more straightforward than bundling these manually.

I hope you've gotten a good understanding of how to bundle Go Lambda functions by reading this. I know I learned a lot, and this will come in handy as I use Go for Lambdas going forward.

Cover photo by Jesse De Meulenaere on Unsplash

A Gentle Introduction to AWS Lambda

Matt Bacchi — Wed, 02 Feb 2022 14:44:48 +0000

You've probably heard of AWS Lambda and serverless by now. But what is Lambda all about? The short definition of AWS Lambda is a "Functions as a Service" (FaaS) technology. The longer and more complicated answer is that Lambda is a lightweight runtime that requires no infrastructure to be defined by the developer.

FaaS allows developers to build software features quickly with less emphasis on the question of where and how they run. This allows for more focus on the business logic that users interact with. Functions as a Service is the most recent innovation in the long line of virtualization technologies. Compute virtualization started with physical servers, then moved to virtual machines (VMs), which grew into Infrastructure as a Service (IaaS). Platforms as a Service came next with companies like Heroku allowing developers to deploy without caring about infrastructure specifics. Now we're arriving at FaaS, which promises to abstract all these details away. Here are a couple good descriptions of these origins and where we are as an industry today.

Lambda vs. Serverless Terminology

The terms serverless and AWS Lambda are often used interchangeably, but they aren't exactly the same thing. Serverless refers to a collection of tools that require no infrastructure configurations from the developer, including databases, queues, functions and gateways as a service. AWS Lambda is only the FaaS component in this list. That means there are many more products such as API gateways, message queueing systems, notification tools, and serverless databases that fill out the serverless landscape. In this blog post we'll only work with a couple of these, namely AWS Lambda and AWS API Gateway for our demonstration.

FaaS Platforms

There are many FaaS providers in the industry. From the big players like Google, IBM, and AWS to the smaller ones like the open source OpenWhisk and OpenFaaS, they all provide similar functionality (generally) with different developer experiences.

There are also Edge computing FaaS providers, which instead of executing the code in one central data center, run the code in a data center nearest the user, known as the "Edge". The players in this area of serverless technology are companies like Cloudflare, Lambda@Edge (AWS), and Fastly.

AWS Lambda in Plan English

After reading the above definition of serverless, Functions as a Service, and AWS Lambda, your understanding might still be cloudy (no pun intended). I'll attempt to explain it in more simple terms.

The description on the AWS documentation website says "Lambda is a compute service that lets you run code without provisioning or managing servers." The essence of Lambda is that AWS built all the scaffolding to allow you and I to send them a single piece of code, which they will run for us.

We don't need to build servers or container images in order to run this single piece of code. All we have to think about is writing code, and then a small SAM template file which defines the requirements of the deployment, and finally shipping it.

Is it Really a Single Function?

In order to wrap your brain around this idea of functions as a service, think of a single method or function in a small program. The function is called by another function, and may return a result.

Below you see a simple Python program in which there is a parent function (main()) calling a child function (childfunction()).

def childfunction(inputstring):
    if inputstring == "hello":
        return "world"
    return "hi"


def main():
    value = childfunction("hello")
    print(f"value: {value}")


if __name__ == "__main__":
    main()

The output of this program looks like this:

$ python hello.py
value: world

At it's simplest form, AWS Lambda can be thought of as a single child function being called by a parent function. AWS owns the parent function, which executes your function (the child function) in it's own virtual environment.

Your objective during the rest of this blog post is to write, deploy and execute a single Lambda function.

Let's Write One Bite Sized Function

For this example, we'll write a small function that does nothing particularly important, but demonstrates the mechanics of running a function in AWS Lambda. You are probably familiar with shell environment variables. In your terminal you can print out the environment variables defined in your session, or even create new environment variables.

The below terminal command prints out an environment variable showing the user name that you are logged in as:

$ echo $USER
mbacchi

What Will the Function Do?

The function we'll deploy will print out the default runtime environment variables in the AWS Lambda execution environment.

Similar to the terminal session above, our Lambda function can access the default environment variables such as AWS_LAMBDA_FUNCTION_NAME, AWS_LAMBDA_LOG_GROUP_NAME, or TZ. But how do we see the output of print statements in a Lambda? Anything that gets printed to STDOUT will be logged to CloudWatch Logs using the log group name of the function.

How Does Lambda Work?

The concepts that you'll need to understand are described in the AWS Lambda documentation.

Here are some of the most important details you'll need to write your Lambda functions:

Invocation: The function is invoked by one of the triggers that AWS provides. It can also be triggered by cron like scheduling events.
Entrypoint: Your Lambda function needs to implement a handler which is the method (or function) that gets called by AWS when triggered.
Environment: The handler is passed an event object and context object from AWS that provides your function both data to be processed and context (information) about how it was called and other clues that you can use in your code.
Response: In certain cases you will want to syncronously return a value from your function, but that is optional. If you are writing an API backend you will return a value through API Gateway. If instead you are performing some scheduled data processing you may not return a value.

Background on Associated Serverless Components

We mentioned earlier how serverless isn't just AWS Lambda, and in this project we'll use API Gateway and CloudWatch Logs, but only to enable us to show how our Lambda function executes.

AWS API Gateway is, as the name suggests, an API gateway product. But it also allows for more than just API calls. For example it enables generic HTTP/HTTPS traffic, REST APIs and Websocket APIs. You can host a standard web server with API Gateway and Lambda, you aren't limited to just APIs with API Gateway.

CloudWatch Logs is a logging product from AWS that allows all other AWS products to log their activity to a central location. In our example we'll be looking at the output of our Lambda function to see the environment variables available in the Lambda itself during runtime.

We're not going to cover these in any more detail because they're not the main focus of this post.

Sample Lambda Function

For this example deployment, we'll use a GitHub repository that has our AWS Lambda function defined. The Lambda function in this repo was written with the intention of being used for demonstration purposes, and as a simple function that can be deployed when testing out CI/CD or the many serverless deployment tools available today (such as Serverless Framework, Cloud Development Kit (CDK) etc.)

Deploying a Lambda Function with AWS SAM

AWS SAM is the Serverless Application Model, which allows you to write a template describing the Lambda function, and any related resources you need to run your Lambda. The SAM template uses the YAML markup language and is based on AWS Cloudformation. After reading the template, SAM performs the deployment for you, and you can watch Cloudformation events to identify what the outcome was.

We are going to use the template in the above repository to deploy the function. I won't go into detail breaking down all of the items in the template for now. That information can be found in the documentation above or in other blogs. But I do want to highlight the two resources used here, the LambdaEnvVarsFunction which is of the type AWS::Serverless::Function and the LambdaLogGroup, which is the definition of our AWS::Logs::LogGroup CloudWatch log group, where our Lambda output will be collected.

Install SAM CLI

You'll need to install a few components before running SAM. I would recommend running this in a Python virtual environment (venv), and I will provide basic steps here to set that up.

From your terminal run these commands (this is on a Linux system):

Step 1: Make sure you have Python 3.8 (you may have to install this first, on Fedora Linux the command would be sudo dnf install python3.8)

  $ which python3.8
  /usr/bin/python3.8

Step 2: Create a Python 3.8 virtual environment

  $ python3.8 -m venv venv38
  $ ls venv38
  bin  include  lib  lib64  pyvenv.cfg

Step 3: Activate the virtual environment

  $ . venv38/bin/activate
  $ which python
  ~/FAKE_PATH/lambda-env-vars7/venv38/bin/python
  $ which pip
  ~/FAKE_PATH/lambda-env-vars7/venv38/bin/pip

Step 4: Install SAM CLI NOTE: This installs Docker in the Python virtual environment

  $ pip install aws-sam-cli
  Collecting aws-sam-cli
    Using cached aws_sam_cli-1.37.0-py3-none-any.whl (5.0 MB)
  Collecting boto3==1.*,>=1.18.32
    Using cached boto3-1.20.46-py3-none-any.whl (131 kB)
  ...

Now you can move on to the next step, where we run the Lambda function locally.

Running Your Lambda Function with SAM Local

Before we deploy to AWS itself, we want to test our code by invoking the Lambda function locally. What is this magic you ask? SAM provides a utility command called sam local invoke which allows your function to run locally in a Docker container to test that it runs as expected before deploying to AWS.

NOTE: We assume you haven't left the terminal session in step 4 above. If you have, activate your Python virtual environment as in step 3 above.

Run sam local invoke as described in the Git repository (this uses test events supplied with Lambda function in the repository)

$ sam local invoke LambdaEnvVarsFunction --event events/event.json
Invoking app.lambda_handler (python3.8)
Skip pulling image and use local one: public.ecr.aws/sam/emulation-python3.8:rapid-1.37.0-x86_64.

Mounting /home/mbacchi/data/repos/mbacchi/lambda-env-vars7/.aws-sam/build/LambdaEnvVarsFunction as /var/task:ro,delegated inside runtime container
START RequestId: f97086cf-e8b6-4d9f-b2bc-b627ca5961e6 Version: $LATEST
Hello there this is the body: {'_HANDLER': 'app.lambda_handler', 'AWS_REGION': 'us-east-1', 'AWS_EXECUTION_ENV': 'AWS_Lambda_python3.8', 'AWS_LAMBDA_FUNCTION_NAME': 'LambdaEnvVarsFunction', 'AWS_LAMBDA_FUNCTION_MEMORY_SIZE': '128', 'AWS_LAMBDA_FUNCTION_VERSION': '$LATEST', 'AWS_LAMBDA_LOG_GROUP_NAME': 'aws/lambda/LambdaEnvVarsFunction', 'AWS_LAMBDA_LOG_STREAM_NAME': '$LATEST', 'LANG': 'en_US.UTF-8', 'TZ': ':/etc/localtime', 'LAMBDA_TASK_ROOT': '/var/task', 'LAMBDA_RUNTIME_DIR': '/var/runtime', 'PATH': '/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin', 'LD_LIBRARY_PATH': '/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib', 'PYTHONPATH': '/var/runtime', 'AWS_LAMBDA_RUNTIME_API': '127.0.0.1:9001'}
END RequestId: f97086cf-e8b6-4d9f-b2bc-b627ca5961e6
REPORT RequestId: f97086cf-e8b6-4d9f-b2bc-b627ca5961e6  Init Duration: 0.11 ms  Duration: 100.72 ms Billed Duration: 101 ms Memory Size: 128 MB Max Memory Used: 128 MB 
{"statusCode": 200, "body": "{\"_HANDLER\": \"app.lambda_handler\", \"AWS_REGION\": \"us-east-1\", \"AWS_EXECUTION_ENV\": \"AWS_Lambda_python3.8\", \"AWS_LAMBDA_FUNCTION_NAME\": \"LambdaEnvVarsFunction\", \"AWS_LAMBDA_FUNCTION_MEMORY_SIZE\": \"128\", \"AWS_LAMBDA_FUNCTION_VERSION\": \"$LATEST\", \"AWS_LAMBDA_LOG_GROUP_NAME\": \"aws/lambda/LambdaEnvVarsFunction\", \"AWS_LAMBDA_LOG_STREAM_NAME\": \"$LATEST\", \"LANG\": \"en_US.UTF-8\", \"TZ\": \":/etc/localtime\", \"LAMBDA_TASK_ROOT\": \"/var/task\", \"LAMBDA_RUNTIME_DIR\": \"/var/runtime\", \"PATH\": \"/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin\", \"LD_LIBRARY_PATH\": \"/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib\", \"PYTHONPATH\": \"/var/runtime\", \"AWS_LAMBDA_RUNTIME_API\": \"127.0.0.1:9001\"}"}

Deploy to AWS

NOTE: Before deploying you need to setup your AWS credentials properly.

In order to deploy the Lambda function, we'll follow the steps in the lambda-env-vars repository:

Step 1: Run sam build

  $ sam build
  Building codeuri: /home/mbacchi/data/repos/mbacchi/lambda-env-vars7/env_vars runtime: python3.8 metadata: {} architecture: x86_64 functions: ['LambdaEnvVarsFunction']
  Running PythonPipBuilder:ResolveDependencies
  Running PythonPipBuilder:CopySource

  Build Succeeded

  Built Artifacts  : .aws-sam/build
  Built Template   : .aws-sam/build/template.yaml

  Commands you can use next
  =========================
  [*] Invoke Function: sam local invoke
  [*] Test Function in the Cloud: sam sync --stack-name {stack-name} --watch
  [*] Deploy: sam deploy --guided

Step 2: Run sam package (this creates an S3 bucket, which must be removed later)

  $ sam package --output-template-file packaged.yaml --resolve-s3
    Creating the required resources...
    Successfully created!

      Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-61z4v277mwy0
      A different default S3 bucket can be set in samconfig.toml
      Or by specifying --s3-bucket explicitly.
  Uploading to eb6c7534d67a5b044653e48f2802a548  452695 / 452695  (100.00%)

  Successfully packaged artifacts and wrote output template to file packaged.yaml.
  Execute the following command to deploy the packaged template
  sam deploy --template-file /home/mbacchi/data/repos/mbacchi/lambda-env-vars7/packaged.yaml --stack-name <YOUR STACK NAME>

Step 3: Run sam deploy

  $ sam deploy --template-file packaged.yaml --region us-east-2 --capabilities CAPABILITY_IAM --stack-name lambda-env-vars --resolve-s3

      Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-61z4v277mwy0
      A different default S3 bucket can be set in samconfig.toml
      Or by specifying --s3-bucket explicitly.

    Deploying with following values
    ===============================
    Stack name                   : lambda-env-vars
    Region                       : us-east-2
    Confirm changeset            : False
    Disable rollback             : False
    Deployment s3 bucket         : aws-sam-cli-managed-default-samclisourcebucket-61z4v277mwy0
    Capabilities                 : ["CAPABILITY_IAM"]
    Parameter overrides          : {}
    Signing Profiles             : {}

  Initiating deployment
  =====================
  Uploading to 77524ed205c6d82d5487433831c24c0f.template  1635 / 1635  (100.00%)

  Waiting for changeset to be created..

  CloudFormation stack changeset
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Operation                                                  LogicalResourceId                                          ResourceType                                               Replacement                                              
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  + Add                                                      LambdaEnvVarsFunctionLambdaEnvVarsPermissionProd           AWS::Lambda::Permission                                    N/A                                                      
  + Add                                                      LambdaEnvVarsFunctionRole                                  AWS::IAM::Role                                             N/A                                                      
  + Add                                                      LambdaEnvVarsFunction                                      AWS::Lambda::Function                                      N/A                                                      
  + Add                                                      LambdaLogGroup                                             AWS::Logs::LogGroup                                        N/A                                                      
  + Add                                                      ServerlessRestApiDeploymentcaa6ada684                      AWS::ApiGateway::Deployment                                N/A                                                      
  + Add                                                      ServerlessRestApiProdStage                                 AWS::ApiGateway::Stage                                     N/A                                                      
  + Add                                                      ServerlessRestApi                                          AWS::ApiGateway::RestApi                                   N/A                                                      
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

  Changeset created successfully. arn:aws:cloudformation:us-east-2:592431548397:changeSet/samcli-deploy1643607514/85a31faf-1358-47ff-9eeb-4428020bb482


  2022-01-30 22:38:46 - Waiting for stack create/update to complete

  CloudFormation events from stack operations
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  ResourceStatus                                             ResourceType                                               LogicalResourceId                                          ResourceStatusReason                                     
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  CREATE_IN_PROGRESS                                         AWS::IAM::Role                                             LambdaEnvVarsFunctionRole                                  -                                                        
  CREATE_IN_PROGRESS                                         AWS::IAM::Role                                             LambdaEnvVarsFunctionRole                                  Resource creation Initiated                              
  CREATE_COMPLETE                                            AWS::IAM::Role                                             LambdaEnvVarsFunctionRole                                  -                                                        
  CREATE_IN_PROGRESS                                         AWS::Lambda::Function                                      LambdaEnvVarsFunction                                      -                                                        
  CREATE_IN_PROGRESS                                         AWS::Lambda::Function                                      LambdaEnvVarsFunction                                      Resource creation Initiated                              
  CREATE_COMPLETE                                            AWS::Lambda::Function                                      LambdaEnvVarsFunction                                      -                                                        
  CREATE_IN_PROGRESS                                         AWS::Logs::LogGroup                                        LambdaLogGroup                                             -                                                        
  CREATE_IN_PROGRESS                                         AWS::ApiGateway::RestApi                                   ServerlessRestApi                                          -                                                        
  CREATE_IN_PROGRESS                                         AWS::ApiGateway::RestApi                                   ServerlessRestApi                                          Resource creation Initiated                              
  CREATE_COMPLETE                                            AWS::ApiGateway::RestApi                                   ServerlessRestApi                                          -                                                        
  CREATE_IN_PROGRESS                                         AWS::Logs::LogGroup                                        LambdaLogGroup                                             Resource creation Initiated                              
  CREATE_IN_PROGRESS                                         AWS::ApiGateway::Deployment                                ServerlessRestApiDeploymentcaa6ada684                      -                                                        
  CREATE_IN_PROGRESS                                         AWS::Lambda::Permission                                    LambdaEnvVarsFunctionLambdaEnvVarsPermissionProd           Resource creation Initiated                              
  CREATE_IN_PROGRESS                                         AWS::Lambda::Permission                                    LambdaEnvVarsFunctionLambdaEnvVarsPermissionProd           -                                                        
  CREATE_COMPLETE                                            AWS::Logs::LogGroup                                        LambdaLogGroup                                             -                                                        
  CREATE_IN_PROGRESS                                         AWS::ApiGateway::Deployment                                ServerlessRestApiDeploymentcaa6ada684                      Resource creation Initiated                              
  CREATE_COMPLETE                                            AWS::ApiGateway::Deployment                                ServerlessRestApiDeploymentcaa6ada684                      -                                                        
  CREATE_IN_PROGRESS                                         AWS::ApiGateway::Stage                                     ServerlessRestApiProdStage                                 -                                                        
  CREATE_IN_PROGRESS                                         AWS::ApiGateway::Stage                                     ServerlessRestApiProdStage                                 Resource creation Initiated                              
  CREATE_COMPLETE                                            AWS::ApiGateway::Stage                                     ServerlessRestApiProdStage                                 -                                                        
  CREATE_COMPLETE                                            AWS::Lambda::Permission                                    LambdaEnvVarsFunctionLambdaEnvVarsPermissionProd           -                                                        
  CREATE_COMPLETE                                            AWS::CloudFormation::Stack                                 lambda-env-vars                                            -                                                        
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

  CloudFormation outputs from deployed stack
  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Outputs                                                                                                                                                                                                                                   
  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Key                 LambdaEnvVarsApi                                                                                                                                                                                                      
  Description         API Gateway endpoint URL for Prod stage for Lambda Env Vars function                                                                                                                                                  
  Value               https://e4cusxvy7f.execute-api.us-east-2.amazonaws.com/Prod/env_vars/                                                                                                                                                 

  Key                 LambdaLogGroup                                                                                                                                                                                                        
  Description         Cloudwatch Log Group ARN                                                                                                                                                                                              
  Value               /aws/lambda/lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp                                                                                                                                                        

  Key                 LambdaEnvVarsFunctionIamRole                                                                                                                                                                                          
  Description         Implicit IAM Role created for Lambda Env Vars function                                                                                                                                                                
  Value               arn:aws:iam::592431548397:role/lambda-env-vars-LambdaEnvVarsFunctionRole-L05B77IT7WK0                                                                                                                                 

  Key                 LambdaEnvVarsFunction                                                                                                                                                                                                 
  Description         Lambda Env Vars Function ARN                                                                                                                                                                                          
  Value               arn:aws:lambda:us-east-2:592431548397:function:lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp                                                                                                                     
  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

  Successfully created/updated stack - lambda-env-vars in us-east-2

The sam deploy command has deployed the Lambda function to AWS and returned the API Gateway endpoint which we can use to invoke the Lambda with an API call. We need the API Gateway URL from the above output, look for the LambdaEnvVarsApi key. In our example above, this is:

https://e4cusxvy7f.execute-api.us-east-2.amazonaws.com/Prod/env_vars/

GET Request to Invoke the Lambda Function

Now that we have the API Gateway URL (also known as the endpoint,) we can use a tool like curl to perform an HTTP GET request against the endpoint. Because we defined the Events stanza with a Type of Api, and Method get, SAM has deployed an API Gateway resource to trigger our Function. (More detailed discussion on this implicit API Gateway resource here).

We will use the command curl to perform a GET request as below:

$ curl -X GET https://e4cusxvy7f.execute-api.us-east-2.amazonaws.com/Prod/env_vars/
{"_HANDLER": "app.lambda_handler", "AWS_REGION": "us-east-2", "AWS_EXECUTION_ENV": "AWS_Lambda_python3.8", "AWS_LAMBDA_FUNCTION_NAME": "lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp", "AWS_LAMBDA_FUNCTION_MEMORY_SIZE": "128", "AWS_LAMBDA_FUNCTION_VERSION": "$LATEST", "AWS_LAMBDA_LOG_GROUP_NAME": "/aws/lambda/lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp", "AWS_LAMBDA_LOG_STREAM_NAME": "2022/01/31/[$LATEST]c2b9b9c953154c8daa7c9b2c0637b70e", "LANG": "en_US.UTF-8", "TZ": ":UTC", "LAMBDA_TASK_ROOT": "/var/task", "LAMBDA_RUNTIME_DIR": "/var/runtime", "PATH": "/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin", "LD_LIBRARY_PATH": "/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib", "PYTHONPATH": "/var/runtime", "AWS_LAMBDA_RUNTIME_API": "127.0.0.1:9001"}

On line 2, we can see the response JSON, which is what the
Lambda function returned to us via the API Gateway. And we see all the environment variables that we had expected, just as in the sam local invoke command earlier.

But how can we observe the output of the print statement at line 50?

To do that we look at the CloudWatch log output. Browse to your AWS console, and search for the Cloudwatch log group name that was in the Outputs section of the sam deploy command output.

In our case the LambdaLogGroup name was: /aws/lambda/lambda-env-vars-LambdaEnvVarsFunction-6r0qwkWGeZSp

When I browse there in CloudWatch, I see the "Hello there" line in the output, which was sent to CloudWatch because we performed a print to standard output within the Lambda function code:

What Just Occurred? What's Next?

Congratulations, you just deployed and invoked your first Lambda function.

Take some time to examine the output of the function in CloudWatch Logs, as well as what was returned during your curl command.

Challenge 1: If you've heard of the Postman tool, you can also call the API endpoint from there. Give it a shot.

Challenge 2: You can display the HTTP headers when calling the API endpoint. How would you see the headers with the curl command? How would you see them with Postman?

Challenge 3: Read more about Lambda functions, SAM, Serverless Framework, CDK. Build your own Lambda and deploy it. Let me know how it goes on Twitter @fshwsprr.

Clean Up AWS Resources

When you are done playing around with this Lambda function, run these commands to remove your resources:

aws cloudformation delete-stack --stack-name lambda-env-vars --region us-east-2
aws s3 rb s3://aws-sam-cli-managed-default-samclisourcebucket-61z4v277mwy0 --force NOTE: This requires you to look at the output of your sam package or sam deploy command above to get the name of the S3 bucket to remove!
Finally, you can deactivate your python virtual environment with the command: deactivate

Summary

If you've read this far, the important takeaways about AWS Lambda in my opinion, are:

The concept that Lambda functions are only one part of the cloud serverless landscape.
Understanding invocation options and how your single function gets called by AWS.
The fact that you can locally invoke the function in SAM Local before you deploy to AWS.
Anything you print inside your function will be written to CloudWatch Logs (assuming you created a Log Group for your Lambda.)

Thanks for reading! Enjoy your serverless and Lambda journey!

Cover photo by Avel Chuklanov on Unsplash

How to Avoid CIDR Conflicts in AWS Sagemaker Notebooks

Matt Bacchi — Thu, 13 Jan 2022 15:06:33 +0000

Networking can sometimes be quite complicated. Despite the oft repeated joke that "It's always DNS", sometimes your problem is even more difficult to diagnose than DNS.

According to Wikipedia, Classless Inter-Domain Routing (or CIDR) "is the method for allocating IP addresses and for IP routing" on the internet and on private networks. If there are conflicts in two networks' CIDR ranges, it can cause headaches that make DNS problems look like childs play.

This is a story about how I unknowingly created a CIDR conflict, and I hope it will be useful to help you avoid CIDR conflicts in Sagemaker Notebooks in the future.

What is AWS Sagemaker

AWS Sagemaker is a popular Machine Learning platform that provides preconfigured environments which allow you to begin training ML models quickly.

Sagemaker Notebooks provide the platform to build and train your models. Under the covers an AWS Sagemaker Notebook is an EC2 instance running a Jupyter notebook packaged with numerous libraries and algorithms. This prevents users from having to configure their own compute, storage and libraries themselves.

A benefit of Sagemaker Notebooks is that you can run generic Python commands alongside your more complex ML code within the Jupyter notebooks. For example, while testing out some connectivity issues recently I was able to use the ubiquitous Python Requests library to perform an HTTP GET request within the Sagemaker Notebook to verify I was able to communicate with a webserver I launched in another AWS account.

Creating a VPC

When I create new projects or resources in AWS accounts, I find it useful to create a new VPC just for the resources in that specific project. This provides network isolation, security and connectivity to the specific components needed for the project. During VPC creation you need to provide a CIDR range that the VPC will use.

CIDR ranges must be built using the private address spaces that IETF RFC 1918 dictates. This means that you choose the CIDR range, but it must be in the 192.168.0.0 range for 256 Class C networks, or 172.16.0.0 for 16 Class B networks, or 10.0.0.0 for a Class A network.

The day I was working on these Sagemaker resources, I randomly chose the range 172.17.0.0 for this VPC, and this fateful decision would come to haunt me.

Sagemaker Notebooks

As described briefly above, Sagemaker Notebooks run on an AWS EC2 instance, and provide preconfigured tools for ML use cases. Part of that tooling is a wonderful open source product called Jupyter Notebook,
which is a web based Python experimentation environment. Jupyter Notebooks run inside Docker on the Sagemaker Notebook EC2 instance.

When you configure Docker on an EC2 instance or any workstation/laptop/Linux machine, it sets up virtual networking of its own to provide network connectivity to the Docker containers. This local virtual networking sets up a bridge network interface that allows communication between the Docker network and the local machine as well as the internet if allowed. The default Docker bridge network uses the range 172.17.0.0 for the docker0 network interface.

CIDR Range Conflict Causes Connectivity Problem

Now that the background is set, you can see why a connectivity problem was destined to occur on the Sagemaker Notebook instance. Since the VPC CIDR range utilized 172.17.0.0, that meant all EC2 instances or network interfaces created in that VPC would be provided with an IP address within the 172.17.0.0 range.

Because the Jupyter Notebook running in Docker on the Sagemaker Notebook EC2 instance used the Docker bridge network 172.17.0.0 and listened for all traffic being sent to that destination, it superceded the traffic sent to the
outside world. Any network packets sent to the default route (0.0.0.0) via the default gateway (172.17.112.1) were actually intercepted by the same Docker bridge network, and not sent outside the bridge network.

This is shown via the route -n command on the Sagemaker Notebook EC2 instance terminal:

sh-4.2$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.17.112.1    0.0.0.0         UG    10002  0        0 eth2
169.254.0.0     0.0.0.0         255.255.255.0   U     0      0        0 veth_def_agent
169.254.169.254 0.0.0.0         255.255.255.255 UH    0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.17.112.0    0.0.0.0         255.255.240.0   U     0      0        0 eth2
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-9bb29e923d05
192.168.0.0     0.0.0.0         255.255.0.0     U     0      0        0 bridge0

Was This Unexpected?

What could have been done to avoid this you ask?

As I mentioned earlier, the default Docker bridge network uses the range 172.17.0.0 for the docker0 network interface. It's fine that AWS simply used that default without modifying it, but that implementation detail should have been documented in the Sagemaker Notebook documentation.

Ideally my suggested resolution is different. If the AWS end user (aka customer) tries creating a Sagemaker Notebook in a VPC that has a CIDR range the same as the default Docker bridge range (172.17.0.0), they should change the Docker bridge network range to something else to avoid a conflict.

Wrap Up

Ultimately the responsibility was on me to create the VPC and the Sagemaker Notebook and make sure they worked correctly. But this was made infinitely more difficult by AWS making choices themselves. Because AWS Sagemaker did not document the Docker bridge network range they used, and did not programatically configure an alternate Docker bridge network range if the customer selected the same CIDR range for their VPC as the default Docker bridge network, I was left troubleshooting a nasty problem.

Please be aware this is still an open issue. (I say that in a figurative way not literal. There is no AWS Support ticket open, and they have not committed to fixing this in any way.)

If you want to avoid CIDR conflicts on Sagemaker Notebooks in your AWS VPC, make sure you use a CIDR range other than 172.17.0.0.

DEV Community: Matt Bacchi

Using the VSCode Claude Code Extension with Bedrock and Claude Sonnet 4.5

Overview

Unfortunate Missing Features and a Bug/Regression with the Claude Code Extension

Create IAM User and Bedrock Permissions

Setup AWS Profile

Enable Bedrock Claude Sonnet 4.5 and Validate Access

Install and Configure VSCode

Did it work?

Conclusion

Resources

Firecracker Virtualization Overview

Firecracker High Level Overview

Configuration

Initialization

Enhanced Isolation using Jailer

Demonstration

Firecracker Ecosystem Discrepancies Compared to Lambda

Synchronous

Asynchronous

The Cold Truth

Conclusion

Installing Nvidia Datacenter GPU Manager on Amazon Linux 2023

What is the Datacenter GPU Manager

Amazon Linux 2 End Of Life

Installing DCGM on AL2023

Conclusion

Switching to the Terraform S3 Backend with Native State File Locks

What is the Terraform S3 Backend

What is State File Locking

AWS S3 Conditional Writes

S3 Conditional Write Support Added in Terraform v1.10

Configuring the S3 Backend to Use Native State File Locking

Terraform Configuration Specifics

Version Constraints

Sample Configuration

State File Locking in Action

Error from DynamoDB State File Locking

Error from S3 Backend Native State File Locking

Dealing with Stale State File Locks

Delete Your Old DynamoDB Tables

Summary

A Survey of Serverless Sustainability Trends

What is Sustainability

Sustainable Workloads

Region Selection

Over-provisioned Capacity and Time Shifting

Time Shifting on Kubernetes

Summary

Limitations of AWS EC2 Image Builder Lifecycle Policies

Standard use case

Why so many AMIs?

EC2 Image Builder Lifecycle Policies

Limitations of the current Lifecycle Policies service

Recommended Solutions (Feature Requests)

Conclusion

Event Driven Processing of ip-ranges.json

What is Event Driven Architecture?

Why use EDA for data processing?

SNS notification for ip-ranges.json

Data organization of the ip-ranges.json file

DynamoDB Single Table Design

Lambda Function Overview

AWS CDK configuration

Wrapping Up

Resources

S3 Bucket Takeover Neutralization

What is Hijacking?

Preventing S3 Bucket Takeover

Decommissioning Process

Conclusion

Bundling Go Lambda Functions with the AWS CDK

AWS CDK Deployment Using Typescript

Source Organization

Bundling Go Functions

Verifying the Output

Success

Summary

A Gentle Introduction to AWS Lambda

Lambda vs. Serverless Terminology