DEV Community: Maurice Borgmeier

How to disable page view by default in draw.io

Maurice Borgmeier — Mon, 17 Nov 2025 16:43:15 +0000

I'm using draw.io to create diagrams or as part of workshops to visualize ideas or organize information. I almost never print any of the diagrams I'm creating, but the app defaults to a page view. I find that I naturally gravitate to trying to fit things within the default page size and get a bit irritated when pages are automatically added or removed depending on where I move stuff.

Fortunately, page view can be disabled in the diagram options by unchecking the checkbox in the diagram menu, or via the View menu. Since I do that for pretty much every diagram, I wondered if there's a better way. Changing the default so that new files start with the whole canvas available is not that difficult once you find the right configuration options. Getting there took me a bit, so I'm writing this quick guide for you (and future me).

draw.io allows you to customize your settings from the Extras -> Configuration menu.

Next, you're greeted by this beautiful text input field with a JSON document.

Here, you can customize the default behavior of draw.io as described in the docs: Configure the draw.io editor. The configuration option for page view is of course not called pageViewEnabled or something like that. Instead, we need to set defaultPageVisible to false. If your configuration block is empty, copy and paste the following:

{
    "defaultPageVisible": false
}

Next, click Apply and restart draw.io. Any new documents that you create should now start with the full canvas instead of the page view.

The other options in my configuration increase the size of the shape thumbnails (a godsend on a large screen) and ensure things get pasted where I want them to be.

Anyway, that's it.

— Maurice

If you're getting the following error (or something like it) when clicking apply, you probably added a comma after the last key-value pair in the JSON document, which it doesn't like:

Expected double-quoted property name in JSON at position 105 (line 6 column 1)

Fixing Pipeline Caching issues with Terraform and the AWS provider

Maurice Borgmeier — Mon, 22 Sep 2025 07:53:29 +0000

I've been analyzing and optimizing the performance of our CI/CD pipeline in a current project and encountered some unexpected behavior with Terraform. Since my Googling didn't lead to useful results, I'm writing this to share my experience. I'll explain how I identified the reason why Terraform didn't use the cached providers and how to avoid the underlying problem with platform specific hashes in the Terraform provider lock file.

We're using a private Gitlab instance as the platform to host our code and have a dedicated runner to execute our pipeline. The terraform part of the pipeline is responsible for rolling out code and infrastructure changes and consists of two stages with their own jobs - plan and apply. If you've used Terraform before, you probably already guessed, that the plan job creates a Terraform plan file, i.e., the diff between the current and the target state. The subsequent apply job consumes that and executes the changes (unless something else touched the state in the mean time). Depending on the configuration, the apply is sometimes automated and in other cases manual, which is one of the reasons for separating the two steps.

The pipeline uses images that don't have Terraform installed, so each of the two jobs first installs Terraform and next runs terraform init to set up the backend and providers. Only once these two steps are complete, the environment is fully set up and we can run terraform plan/apply.

This project primarily deploys resources in AWS, so the code depends on the AWS provider. Unfortunately, that one is a bit on the heavier side. The current version of the extracted provider from the Terraform registry weighs in at about 700MB. The zipped form, which is actually downloaded, is about 150-180MB. Downloading this for each job adds quite a bit of overhead. Even on a fast connection this will take at least a few seconds. The decompression may also take a bit of time depending on your build environment.

In our case, the terraform init that includes more than only the AWS provider took about 3-4 minutes without caching. That means about 6-8 minutes of extra time for each run, which is unacceptable. Actually without caching is not fair - we had configured caching for the .terraform directory, it just wasn't working. Well, it was actually working - the directory was cached and restored between runs, it just didn't make a difference, which was odd.

Below is the caching configuration that's close to what we're using. It's attached to both the plan and apply jobs and creates a separate cache for each deployment stage (e.g. dev/prod). We could have included the .terraform.lock.hcl file as part of the cache key here, which defines which provider versions to use, but more on that later.

cache:
    - key: 'terraform-init-$STAGE'
      paths:
          - .terraform

For a long time, I thought that our cache somehow corrupted the files so that the init-command couldn't see them, but after logging in to the runner I could see that the permissions were perfectly fine. The next step was increasing the log level, which I did by setting the TF_LOG environment variable before the init command.

TF_LOG=TRACE terraform init ...

Here's the abbreviated and commented output. I removed the timestamps and some unrelated messages. Yes, we're using a more recent version of the provider now. I suggest you focus on the comments.

// TF scans the local provider dir and finds the expected version
[TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/aws v6.4.0 for linux_amd64 at .terraform/providers/registry.terraform.io/hashicorp/aws/6.4.0/linux_amd64
// TF registers this as a candidate for the aws provider
[TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/aws/6.4.0/linux_amd64 as a candidate package for registry.terraform.io/hashicorp/aws 6.4.0

...
// TF decides to install the SAME provider from the internet
// instead of using the existing binary
[TRACE] providercache.Dir.InstallPackage: installing registry.terraform.io/hashicorp/aws v6.4.0 from https://releases.hashicorp.com/terraform-provider-aws/6.4.0/terraform-provider-aws_6.4.0_linux_amd64.zip

[TRACE] HTTP client GET request to https://releases.hashicorp.com/terraform-provider-aws/6.4.0/terraform-provider-aws_6.4.0_linux_amd64.zip

[DEBUG] Provider signed by 34365D9472D7468F HashiCorp Security (hashicorp.com/security) <security@hashicorp.com>

// It scans the local directory again
[TRACE] providercache.fillMetaCache: scanning directory .terraform/providers

// And finds the binary it has just downloaded
[TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/aws v6.4.0 for linux_amd64 at .terraform/providers/registry.terraform.io/hashicorp/aws/6.4.0/linux_amd64

[TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/aws/6.4.0/linux_amd64 as a candidate package for registry.terraform.io/hashicorp/aws 6.4.0

In a nutshell, Terraform saw the local provider matching the desired version but still decided to download it again. That didn't make a lot of sense to me and still doesn't. Here, I went down a rabbit hole comparing the checksums of the downloaded binaries and trying to figure out if GitLab somehow modified them or their metadata. Of course it didn't - that would be a strange caching implementation - but I had to be sure.

I was able to narrow down the issue further when I decided to run terraform init again as part of the same job. The second run reused the cached version from the first init and completed almost instantly. Taking a closer look at the output led me to realize that the first run was printing something that the second didn't:

Terraform has made some changes to the provider dependency selections recorded in the .terraform.lock.hcl file. Review those changes and commit them to your version control system if they represent changes you intended to make.

I mentioned the .terraform.lock.hcl briefly earlier, now its time to dive a bit deeper into it and its function. The dependency lock file is one of two mechanisms that take part in the decision which exact version of providers to install. At the time of writing this (Terraform v1.13.x), this file keeps track of which exact version of a provider was used and includes hashes to verify that the correct binary is installed.

When you run terraform init, it checks the version constraints on the provider configuration and also the .terraform.lock.hcl. If there's a version of the provider that satisfies the constraints and is already mentioned in the lock file, it will install that one (unless you specify the -upgrade parameter). Otherwise, it will select a version that matches the constraints and update the lock file.

In my case, both the provider config had a version constraint for v6.4.0 and the lock file included the v6.4.0 too. That means I expected it to take the local version from the cache - but it didn't. As a further troubleshooting step, I included the lock file as a job artifact of the plan job. This caused the apply job to use the cache as I expected, so I was onto something. I downloaded the file and took a look at it. Compared to my local version, the pipeline had added another hash value for the AWS provider.

Then it dawned on me. Providers are compiled go binaries and my dev environment is a Windows VM while the build pipeline is powered by Linux containers. When I upgraded to v6.4.0 a few months back, I did that on the Windows machine and apparently it only added the hashes for the Windows version of the provider. In fact, there is a specific terraform providers lock command that you can use to add the hashes of different platforms to the lock file.

I used the following command on my dev VM to add the hashes of the Windows and Linux x86 versions of the providers.

terraform providers lock \
  -platform=windows_amd64 \ # 64-bit Windows
  -platform=linux_amd64     # 64-bit Linux

This command ran for a while, because it's not downloading only the metadata from the registry, it actually downloads both providers, computes the checksums and adds those to the terraform.lock.hcl file. I'm sure for tiny providers this is done in the blink of an eye, but for bigger ones, such as the AWS provider, this takes a while. This issue is not unique to Windows and Linux. Whenever your dev team is using different platforms, you may experience this. Check out the docs I linked for all supported platforms.

Once I committed the updated lock file, the job run times went down significantly and caching worked as I expected. The install and init phases now take less than 20 seconds, which speeds up each pipeline run (unless the caches are deleted).

This experience didn't give me warm fuzzy feelings about the current implementation. It seems like it will always download the binary from the registry to add the checksum. It would be much more efficient to store these checksums as metadata in the registry. This would probably require changes on both the backend and the frontend, though. An alternative would be to at least compute the checksums based on locally available versions of the provider. This may lead to reducing the load on Hashicorp's CDN as well.

— Maurice

Photo by Wolfgang Weiser on Unsplash

Bug in CloudFront's Continuous Deployment Feature

Maurice Borgmeier — Fri, 12 Sep 2025 08:52:02 +0000

This blog post was inspired by a question on stackoverflow. The user experienced intermittent HTTP 500 error codes from CloudFront. They seemed confident, that their setup was correct, so I was intrigued.

The user had deployed a static website to S3 and was using CloudFront in a continuous deployment configuration. That's a setup, where you have two distributions - production and staging. In such a setup, you can test configuration changes in the staging distribution and divert a fraction of production traffic to it in order to to see how it behaves. Once you're satisfied with your configuration changes in the staging environment, you promote them to the production distribution and serve all traffic from there.

This is a pretty neat feature that allows you to test changes on a subset of real users. You can configure it to send traffic to the staging distribution based on either a header value or a percentage of total traffic (weighted). In the latter case you can additionally enable sticky sessions to ensure that users are typically routed to the same distribution for a consistent experience. There are some constraints if you want to use continuous deployments, but in general it's quite a useful feature.

Back to our original problem from stackoverflow. The user was experiencing random HTTP 500 responses from this setup when using weighted routing. Header-based routing worked perfectly fine, so an underlying permission issue seemed unlikely.

I recreated the setup in one of my accounts and tried to reproduce the issue, which failed - at first. For me everything seemed to work. The next day, the user added a crucial detail - they were using custom error responses. That feature allows you to replace CloudFront's error response with your own and can be used to change the HTTP Status code or serve a prettier error page. Once I enabled custom error pages, I started seeing HTTP 500 codes when I accessed a path that would trigger the error condition (e.g. a 404/403 error) - but not all the time. Here's an example.

We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner. If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.

So, how do we identify what's going on here? Intermittent errors are some of the most annoying to debug. The first step is making sure that we can identify where our response is coming from to figure out if it's related to only one of the distributions, so I created two response header policies that set the Environment header to Production or Staging depending on which distribution served the request.

Through some manual trial and error I found out that the error is only triggered when we request a URL that triggers the custom error response. I wanted to estimate how frequently this happens and under which conditions. I created a configuration for the load testing tool artillery to automate part of this analysis and wrote some custom code to count the responses per distribution.

# load_test.yml
config:
  target: "https://d2dge64jsf7e3f.cloudfront.net"
  phases:
    - duration: 120
      arrivalRate: 50
      name: "Load test phase"
  processor: "./hooks.js"
  plugins:
    metrics-by-endpoint: {}

scenarios:
    # Requesting a non-existent page from the S3 origin triggers an HTTP 403
    # from S3, which should be turned into a HTTP 404 + custom error page by
    # the custom error repsponse config.
  - name: "Non-existent page"
    weight: 100
    flow:
      - get:
          url: "/non-existent-page"
          afterResponse: "logAndMetrics"

This configuration will request a non-existent page 50 times per second for a period of two minutes. It will evaluate each response with the logAndMetrics function from the processor, which is implemented as follows:

// hooks.js
module.exports = {
  logAndMetrics: function(requestParams, response, context, event, next) {
    if (response.statusCode === 500) {
    //   console.log(`Path: ${requestParams.url}`);
      console.log('Headers:', response.headers);
      console.log('Body:', response.body);
    }

    const environment = response.headers.environment || "unknown"
    // Increment counters for the environment and the environment + status code
    event.emit('counter', `environment_${environment}_${response.statusCode}`, 1)
    event.emit('counter', `environment_${environment}`, 1)
    return next();
  }
};

I chose to test with a continuous deployment policy that sends 15% of all requests to the staging distribution, which is the maximum that's supported. This means for any load test I'll get fewer responses from the staging distribution, giving me less confident estimates, but such is life.

resource "aws_cloudfront_continuous_deployment_policy" "weighted" {
  enabled = true

  staging_distribution_dns_names {
    items    = [aws_cloudfront_distribution.staging_distribution.domain_name]
    quantity = 1
  }

  traffic_config {
    type = "SingleWeight"
    single_weight_config {
      weight = "0.15"
    }
  }
}

The code for this analysis is available on Github and deploys two distributions in a continuous deployment configuration with a weighted continuous deployment policy that forwards 15% of the production traffic to the staging distribution. Each distribution has its own response headers policy that allows us to identify which distribution sent the response. They use the same S3 bucket and origin access control as the origin.

In this setup, I tested five permutations - well I planned to test four, but retesting during peak traffic hours changed the behavior, more on that in a bit. I enabled and disabled the custom error responses on both distributions until I had all four permutations and measured the fraction of errors for each configuration. The numbers are rounded a bit and we have less data for the staging distribution as explained above.

Custom Error Enabled (Production)	Custom Error Enabled (Staging)	HTTP 500 (Production)	HTTP 500 (Staging)
Yes	Yes	~15%	~83%
Yes	No	None	~47%
No	Yes	~13%	None
No	No	None	None
Yes	No	None (Peak Traffic)	None (Peak Traffic)

As you can see from the table, enabling it on one distribution influences the other and the biggest impact is seen when it's enabled on both the production and staging environment. I've tried the same tests with sticky sessions enabled, it didn't meaningfully change the numbers, so I assume the underlying issue is independent of that feature. The time of day, though, changed things a bit.

One of the limitations of continuous deployment distributions is, that CloudFront will ignore the configuration during peak traffic hours and stop forwarding traffic to the staging distribution. Under those conditions (like a Friday or Saturday evening when everyone is chilling on the couch) everything was fine, although the same configuration led to a significant number of errors during non-peak hours.

I also confirmed that this is only related to requests that trigger the error behavior. Requesting a known-good URL works all the time. Additionally I tried separate Origin Access Controls, but that didn't change anything.

In summary, I'm a bit confused what's going on here but it was interesting to play around with artillery again.

— Maurice

I have submitted this bug report to AWS and the team was able to reproduce it. I assume this is going to be fixed at some point.

Code on Github

Managing Lambda@Edge Service Quotas

Maurice Borgmeier — Wed, 03 Sep 2025 16:53:44 +0000

Lambda@Edge allows you to run your own business logic as part of the CloudFront request flow. You can intercept all requests that arrive at CloudFront, all connections to the origin and the responses from the origin and to the client. Lambda@Edge functions come with a series of constraints that make them different from regular Lambda functions, but also a lot of commonalities. One of these is the concurrency quota, which you should check before a production deployment.

Lambda@Edge is executed in so-called regional edge caches, which are, at the time of writing this, 13 AWS regions around the world. I explored this in a previous blog: Which AWS Regions are Lambda@Edge functions executed in?

When a request enters an edge location, it contacts its upstream regional edge cache and runs the Lambda functions there. Lambda@Edge ensures that the functions are replicated to all regional edge caches. With Lambda@Edge as part of the request flow, you're typically limited in the amount of requests your distribution can handle, because edge functions share the regional concurrency limits with the regular Lambda functions in that region.

AWS advertises a default limit of 1000 concurrent lambda execution contexts per region in most accounts, but in reality new accounts are frequently throttled to much smaller numbers than that. I've seen plenty of examples, where an account starts out with 10 concurrent execution contexts per region, which is then gradually increased.

That situation is less than ideal if you're trying to deploy a new CloudFront distribution that caters to a global audience. If you receive more concurrent requests than your quota allows, your users will get an HTTP 503 error code, with a response that looks something like this:

503 ERROR

The request could not be satisfied.

The Lambda function associated with the CloudFront distribution was throttled. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.

If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.

The solution is relatively straightforward, though. You need to create a quota increase request in each of the 13 regional edge cache regions. The quota that needs to be increased is called Concurrent Executions with the code L-B99A9384. Since doing this 13 times is a bit tedious, I've asked Claude to help out.

Here's a script that I reviewed, which uses the API to list the current quota values across all regions that are also regional edge caches. To run it, you should ensure that your current shell is configured to connect to your target account, e.g. through the AWS_PROFILE environment variable or setting the credentials directly. It uses the GetServiceQuota-API to get the current values for each region.

#!/bin/bash

# Script to check Lambda concurrent execution quota (L-B99A9384) across regions
# Requires AWS CLI configured with appropriate permissions

# Define regions to check
REGIONS=(
    "ap-south-1"      # Asia Pacific (Mumbai)
    "ap-northeast-2"  # Asia Pacific (Seoul)
    "ap-southeast-1"  # Asia Pacific (Singapore)
    "ap-southeast-2"  # Asia Pacific (Sydney)
    "ap-northeast-1"  # Asia Pacific (Tokyo)
    "eu-central-1"    # EU (Frankfurt)
    "eu-west-1"       # EU (Ireland)
    "eu-west-2"       # EU (London)
    "sa-east-1"       # South America (Sao Paulo)
    "us-east-1"       # US East (N. Virginia)
    "us-east-2"       # US East (Ohio)
    "us-west-1"       # US West (N. California)
    "us-west-2"       # US West (Oregon)
)

# Service quota code for Lambda concurrent executions
QUOTA_CODE="L-B99A9384"
SERVICE_CODE="lambda"

echo "Checking Lambda Concurrent Execution Quotas"
echo "==========================================="
echo "Quota Code: ${QUOTA_CODE}"
echo ""

# Function to check quota in a region
check_quota() {
    local region=$1
    echo -n "Region: ${region} - "

    # Get the quota value
    quota_value=$(aws service-quotas get-service-quota \
        --region "${region}" \
        --service-code "${SERVICE_CODE}" \
        --quota-code "${QUOTA_CODE}" \
        --query 'Quota.Value' \
        --output text 2>/dev/null)

    if [ $? -eq 0 ] && [ "${quota_value}" != "None" ]; then
        printf "%.0f\n" "${quota_value}"
    else
        echo "ERROR (check permissions/availability)"
    fi
}

# Check each region
for region in "${REGIONS[@]}"; do
    check_quota "${region}"
done

It's output will look something like this:

$ ./check_lambda_quota.sh
Checking Lambda Concurrent Execution Quotas
===========================================
Quota Code: L-B99A9384

Region: ap-south-1 - 10
Region: ap-northeast-2 - 10
Region: ap-southeast-1 - 10
Region: ap-southeast-2 - 10
Region: ap-northeast-1 - 10
Region: eu-central-1 - 1000
Region: eu-west-1 - 10
Region: eu-west-2 - 10
Region: sa-east-1 - 10
Region: us-east-1 - 10
Region: us-east-2 - 10
Region: us-west-1 - 10
Region: us-west-2 - 10

As you can see, for most regions the quota values need to be increased to handle a lot of traffic. To do that, you can use the following script. It has a TARGET_QUOTA variable, which it will request as the new quota should the current value be below it. There's also a DRY_RUN variable that you can set to "true" if you just want to see what it would do. Once you run it and you're satisfied with it's proposed actions, you can set DRY_RUN="false" and re-run the script.

It will then use the RequestServiceQuotaIncrease API to trigger the quota increases. You can safely re-run this script multiple times, because that API will prevent you from having multiple concurrent quota increase requests for the same quota in the same region. In that case you'll see an appropriate message in the output.

#!/bin/bash

# Script to check Lambda concurrent execution quota and request increase to 1000 if below threshold
# Requires AWS CLI with servicequotas:GetServiceQuota and servicequotas:RequestServiceQuotaIncrease permissions

# Define regions to check
REGIONS=(
    "ap-south-1"      # Asia Pacific (Mumbai)
    "ap-northeast-2"  # Asia Pacific (Seoul)
    "ap-southeast-1"  # Asia Pacific (Singapore)
    "ap-southeast-2"  # Asia Pacific (Sydney)
    "ap-northeast-1"  # Asia Pacific (Tokyo)
    "eu-central-1"    # EU (Frankfurt)
    "eu-west-1"       # EU (Ireland)
    "eu-west-2"       # EU (London)
    "sa-east-1"       # South America (Sao Paulo)
    "us-east-1"       # US East (N. Virginia)
    "us-east-2"       # US East (Ohio)
    "us-west-1"       # US West (N. California)
    "us-west-2"       # US West (Oregon)
)

QUOTA_CODE="L-B99A9384"
SERVICE_CODE="lambda"
TARGET_QUOTA=1000

# Set to "true" to actually submit requests, "false" for dry-run
DRY_RUN="false"

echo "Lambda Concurrent Execution Quota Increase Script"
echo "================================================="
echo "Target quota: ${TARGET_QUOTA}"
echo "Dry run mode: ${DRY_RUN}"
echo ""

# Function to check and potentially request quota increase
process_region() {
    local region=$1
    echo -n "Region: ${region} - "

    # Get current quota value
    quota_value=$(aws service-quotas get-service-quota \
        --region "${region}" \
        --service-code "${SERVICE_CODE}" \
        --quota-code "${QUOTA_CODE}" \
        --query 'Quota.Value' \
        --output text 2>/dev/null)

    if [ $? -ne 0 ] || [ "${quota_value}" == "None" ]; then
        echo "ERROR (unable to retrieve quota)"
        return 1
    fi

    current_quota=$(printf "%.0f" "${quota_value}")
    echo -n "Current: ${current_quota} - "

    if [ "${current_quota}" -lt "${TARGET_QUOTA}" ]; then
        echo -n "NEEDS INCREASE - "

        if [ "${DRY_RUN}" == "true" ]; then
            echo "Would request increase to ${TARGET_QUOTA} (DRY RUN)"
        else
            # Submit quota increase request
            request_output=$(aws service-quotas request-service-quota-increase \
                --region "${region}" \
                --service-code "${SERVICE_CODE}" \
                --quota-code "${QUOTA_CODE}" \
                --desired-value "${TARGET_QUOTA}" \
                --query 'RequestedQuota.Id' \
                --output text 2>&1)

            request_exit_code=$?

            if [ $request_exit_code -eq 0 ] && [ "${request_output}" != "None" ]; then
                echo "Request submitted (ID: ${request_output})"
            elif echo "${request_output}" | grep -q "ResourceAlreadyExistsException"; then
                echo "Request already pending"
            elif echo "${request_output}" | grep -q "InvalidParameterValueException"; then
                echo "Invalid request (quota may already be >= ${TARGET_QUOTA})"
            else
                echo "FAILED to submit request: ${request_output}"
            fi
        fi
    else
        echo "OK (already >= ${TARGET_QUOTA})"
    fi
}

# Process each region
for region in "${REGIONS[@]}"; do
    process_region "${region}"
done

echo ""
if [ "${DRY_RUN}" == "true" ]; then
    echo "To submit actual requests, change DRY_RUN to 'false'"
    echo ""
fi

This is what the output could look like.

$ ./increase_lambda_quota.sh 
Lambda Concurrent Execution Quota Increase Script
=================================================
Target quota: 1000
Dry run mode: false

Region: ap-south-1 - Current: 10 - NEEDS INCREASE - Request submitted (ID: ...)
Region: ap-northeast-2 - Current: 10 - NEEDS INCREASE - Request submitted (ID: ...)
Region: ap-southeast-1 - Current: 10 - NEEDS INCREASE - Request submitted (ID: ...)
Region: ap-southeast-2 - Current: 10 - NEEDS INCREASE - Request submitted (ID: ...)
Region: ap-northeast-1 - Current: 10 - NEEDS INCREASE - Request submitted (ID: ...)
Region: eu-central-1 - Current: 1000 - OK (already >= 1000)
Region: eu-west-1 - Current: 10 - NEEDS INCREASE - Request submitted (ID: ...)
Region: eu-west-2 - Current: 10 - NEEDS INCREASE - Request already pending
Region: sa-east-1 - Current: 10 - NEEDS INCREASE - Request submitted (ID: ...)
Region: us-east-1 - Current: 10 - NEEDS INCREASE - Request already pending
Region: us-east-2 - Current: 10 - NEEDS INCREASE - Request already pending
Region: us-west-1 - Current: 10 - NEEDS INCREASE - Request submitted (ID: ...)
Region: us-west-2 - Current: 10 - NEEDS INCREASE - Request submitted (ID: ...)

Once these requests are created, it's time to wait. I've found that the best way to monitor progress is through the list of support cases - the service quotas console shows only regional requests and clicking through all regions is a bit annoying.

It can sometimes take multiple days for AWS to increase the quotas, so I suggest you plan ahead before your go-live.

That's it, I hope this will help you! If you want to learn a lot more about CloudFront and HTTP Caching, check out my course on Udemy in which we dive a lot deeper!

Cheers

— Maurice

Cover Photo by Lee Tianxian on Unsplash

Building a Stream Deck plugin to invoke a Lambda function

Maurice Borgmeier — Fri, 20 Jun 2025 12:46:07 +0000

Interacting with Cloud services is rarely a tactile experience. You write some code, run some command or click a button on a screen and things happen. Today we're going to change that. We'll write a plugin for the Elgato Stream Deck to trigger an AWS Lambda function on demand with a customizable event.

In case you haven't heard of it, the Stream Deck is basically a set of buttons with tiny screens behind them that you can customize to do your bidding through plugins. Today, we'll write a small plugin that invokes a Lambda function of our choosing, thereby allowing us to do pretty much anything in AWS.

My first attempt at this a couple of years ago was not very successful. Since then I've gained more Typescript experience and the state of the Stream Deck SDK has improved substantially, so I gave it another shot.

Let's dive into it. Stream Deck plugins are small Javascript / Typescript apps that talk to the Stream Deck app via a Websocket connection. This means they mostly respond to events such as a button being pressed or an action being added to a Stream Deck. They can also initiate actions or monitor system state and respond to that, but for now we'll stick to simple interaction patterns. Writing plugins isn't that hard if you're a bit familiar with the Typescript ecosystem.

You can find the code for this project here on Github.

For this project, I'm going to assume that you have the AWS CLI configured (i.e. credentials are already prepared) and NodeJS installed (I'm using v24.2.0, minimum is version 20). Also, you need a Stream Deck and ideally VS Code. The prerequisites are also covered in the documentation. Next, we're going to install the Stream Deck CLI, which provides a scaffolding for our plugin and helps with the Stream Deck integration.

npm install -g @elgato/cli

Once that's complete, we're ready to create our plugin using streamdeck create. I'm going to call mine AWS Lambda Invoke with the UUID com.mauricebrg.lambda-invoke - you should probably use your own Domain or UUID to avoid conflicts.

$ streamdeck create
 ___ _                        ___         _   
/ __| |_ _ _ ___ __ _ _ __   |   \ ___ __| |__
\__ \  _| '_/ -_) _` | '  \  | |) / -_) _| / /
|___/\__|_| \___\__,_|_|_|_| |___/\___\__|_\_\

Welcome to the Stream Deck Plugin creation wizard.

This utility will guide you through creating a local development environment for a plugin.
For more information on building plugins see https://docs.elgato.com.

Press ^C at any time to quit.

✔ Author: MauriceBrg
✔ Plugin Name: AWS Lambda Invoke
✔ Plugin UUID: com.mauricebrg.lambda-invoke
✔ Description: Invokes Lambda functions from the Stream Deck

✔ Create Stream Deck plugin from information above? Yes

Creating AWS Lambda Invoke...
✔ Enabling developer mode
✔ Generating plugin
✔ Installing dependencies
✔ Building plugin
✔ Finalizing setup

Successfully created plugin!

✔ Would you like to open the plugin in VS Code? Yes

When that's done, it should open VS Code and the folder structure that you'll see looks something like this. The wizard has helpfully provided an example action that implements a counter. This shows us how to lay out our code. I've added some comments to explain what's located where.

$ tree --gitignore -I node_modules
.
├── com.mauricebrg.lambda-invoke.sdPlugin
│   ├── bin    # Content gets regenerated automatically
│   │   ├── package.json
│   │   └── plugin.js
│   ├── imgs   # (Image) Assets for use by the plugin
│   │   ├── actions
│   │   │   └── counter
│   │   │       ├── icon.png
│   │   │       ├── icon@2x.png
│   │   │       ├── key.png
│   │   │       └── key@2x.png
│   │   └── plugin
│   │       ├── category-icon.png
│   │       ├── category-icon@2x.png
│   │       ├── marketplace.png
│   │       └── marketplace@2x.png
│   ├── manifest.json  # Describes the actions we provide
│   └── ui             # UI for the action settings
│       └── increment-counter.html  # Sample action settings
├── package-lock.json
├── package.json       # Defines our package, dependencies & scripts
├── rollup.config.mjs  # Configuration for the bundler
├── src
│   ├── actions
│   │   └── increment-counter.ts  # Sample action implementation
│   └── plugin.ts  # Configuration for the plugin
└── tsconfig.json

Before we start customizing things, I should point out that this setup comes with two scripts prepared for us:

npm run build to turn our Typescript code into Javascript so it can be used by the Stream Deck app
npm run watch which executes the build command once we change anything in the code

You can already go ahead and run npm run watch in a separate Terminal session. If you open the Stream Deck app, you should already see a new action, which you can add to your Stream Deck and interact with.

Next, we'll add the new folder com.mauricebrg.lambda-invoke.sdPlugin/imgs/actions/lambda and place an SVG with the Lambda logo from the official AWS Icon pack there. This allows us to reference a custom image later on. Now we'll start some preparations for us to invoke the Lambda function. We'll need to install the AWS SDK for that, but that requires another plugin for the bundler, since rollup seems to treat the AWS SDK as an ES Module, which it isn't. To work around that, we'll install the JSON plugin for rollup using npm i --save-dev @rollup/plugin-json and edit rollup.config.mjs:

// rollup.config.mjs
import json from "@rollup/plugin-json";
// ...

const config = {
    output: {
        // Other settings
        inlineDynamicImports: true
    }
    // ...
    plugins: [
        json(),
        // Other plugins
    ]
}

Now we can run npm i --save @aws-sdk/client-lambda to install the AWS SDK without further issues. Okay, all of this has been in preparation for our new action. We'll start by defining the code. Create a new file src/actions/invoke-async.ts with this content:

// src/actions/invoke-async.ts
import streamDeck, { action, KeyDownEvent, SingletonAction } from "@elgato/streamdeck";

const logger = streamDeck.logger

@action({UUID: "com.mauricebrg.lambda-invoke.invoke-async"})
export class InvokeAsync extends SingletonAction<InvokeAsyncSettings> {

    // This function is called when the button is pressed
    public override async onKeyDown(ev: KeyDownEvent<InvokeAsyncSettings>): Promise<void> {
        logger.info(ev)
    }

}

type InvokeAsyncSettings = {
    lambdaSettings?: LambdaInvocationSettings
}

type LambdaInvocationSettings = {
    profile: string // Which SDK Creds to use
    region: string // Region that Lambda is located in
    functionName: string // Name of the function to execute
    event: string // Payload to send to Lambda
}

The code defines a class which extends the SingletonAction from the Stream Deck SDK. Additionally it implements a handler for the onKeyDown event that is emitted whenever the key is pressed. At this point we only log the event. You can also find some data structures that define which settings we'll eventually need for this to work. We need to configure which SDK config to use to talk to AWS and in what region to invoke which Lambda with which event.

At this point our code is not executed. For that to happen, we need to extend the plugin.ts to register our action as well as the manifest.

// src/plugin.ts

// ...
import { InvokeAsync } from "./actions/invoke-async";

// ...
streamDeck.actions.registerAction(new InvokeAsync())

// ... (insert above before this block)
// Finally, connect to the Stream Deck.
streamDeck.connect();

This tells the Stream Deck where to find the backend code and in the manifest we tell it metadata about our action.

// com.mauricebrg.lambda-invoke.sdPlugin/manifest.json
{
    // ...
    "Actions": [
        {
            "Name": "Async Invocation",
            "UUID": "com.mauricebrg.lambda-invoke.invoke-async",
            "Icon": "imgs/actions/lambda/lambda",
            "Tooltip": "Invokes a Lambda Function asynchronously",
            "PropertyInspectorPath": "ui/increment-counter.html",
            "Controllers": [
                "Keypad"
            ],
            "States": [
                {
                    "Image": "imgs/actions/lambda/lambda",
                    "TitleAlignment": "middle"
                }
            ]
        },
        //... other actions
    ]
    // ...
}

If you look closely, you can see the references to the lambda.svg we created earlier in the Icon and States.Image keys. You should now be able to see new action in the UI. Place it on the Stream Deck and press the button.

Pressing the button doesn't to anything in the UI (yet) and the settings also don't match what we're trying to do - we'll fix both of those in a minute. First, let's look at the logs, because there our action was registered. I've abbreviated the logs a bit so it's not completely overwhelming.

// com.mauricebrg.lambda-invoke.sdPlugin/logs/com.mauricebrg.lambda-invoke.0.log

TRACE Connection: {"action":"com.mauricebrg.lambda-invoke.invoke-async","context":"...","device":"...","event":"keyDown","payload":{"coordinates":{"column":2,"row":2},"isInMultiAction":false,"settings":{}}}

INFO  {"type":"keyDown","action":{"controllerType":"Keypad","device":{"id":"..."},"id":"...","manifestId":"com.mauricebrg.lambda-invoke.invoke-async","coordinates":{"column":2,"row":2},"isInMultiAction":false},"payload":{"coordinates":{"column":2,"row":2},"isInMultiAction":false,"settings":{}}}

TRACE Connection: {"action":"com.mauricebrg.lambda-invoke.invoke-async","context":"...","device":"...","event":"keyUp","payload":{"coordinates":{"column":2,"row":2},"isInMultiAction":false,"settings":{}}}

The log level is currently set to TRACE in src/plugin.ts, which lets us see all communication between the Stream Deck and our Plugin. We can see that two events have happened, first a keyDown and last a keyUp event. Additionally we can see which button was pressed on what device through the coordinates and various identifiers. In between we can see our INFO log with the event that our handler received.

Settings are also passed with each event. These are what allow us to go from stateless to stateful. Essentially, settings are a JSON object that is passed to each event listener and can be read and updated from there. Per-action and global (plugin-level) settings exist and today we'll focus on the former. We can provide a UI to interact with the settings an example of which you can see in form of the slider in the screenshot above.

The UI for settings is based on Stream Deck Plugin web components. We can use them to add per-action configuration options. Since we need profile, region, functionName, and event for our action, I've gone ahead and created a template that allows the user to enter this data (in the most basic way possible). Here's an excerpt:

<!-- com.mauricebrg.lambda-invoke.sdPlugin/ui/invoke-async.html -->
<!DOCTYPE html>
<html>

<head lang="en">
    <title>Async Invocation Settings</title>
    <meta charset="utf-8" />
    <script src="https://sdpi-components.dev/releases/v4/sdpi-components.js"></script>
</head>

<body>

    <sdpi-item label="AWS Profile">
    <sdpi-textfield
        setting="lambdaSettings.profile"
        placeholder="default"
        default="default"
        required>
    </sdpi-textfield>
    </sdpi-item>

    <!-- ... -->

    <sdpi-item label="Event">
        <sdpi-textarea
            setting="lambdaSettings.event"
            rows="5"
            showlength
            default="{}"
            required>
        </sdpi-textarea>
    </sdpi-item>

</body>

</html>

Looking at the setting attribute in the Input definitions, you can probably already tell that this is the path where the value of the input is made accessible to the plugin. In the UI, the HTML is rendered like this:

Finally we have all the plumbing in place to add the logic. First, we add a helper function to parse the settings and add default values for anything not explicitly configured. The second function invokes a Lambda asynchronously based on the settings.

import { LambdaClient, InvokeCommand, InvocationType } from "@aws-sdk/client-lambda";

// ...

// Add default values if the settings are not set.
function parseSettingsAndAddDefaults(settings: InvokeAsyncSettings): InvokeAsyncSettings {
    return {
        lambdaSettings: {
            region: settings.lambdaSettings?.region ?? "eu-central-1",
            functionName: settings.lambdaSettings?.functionName ?? "PythonDemo",
            event: settings.lambdaSettings?.event ?? "{}",
            profile: settings.lambdaSettings?.profile ?? "default"
        }
    }
}

// Invoke the Lambda function based on our settings
async function invokeLambdaAsync(config: LambdaInvocationSettings): Promise<number> {
    const client = new LambdaClient({region: config.region, profile: config.profile})
    const command = new InvokeCommand({
        FunctionName: config.functionName,
        InvocationType: InvocationType.Event
    })

    try {
        const response = await client.send(command)
        logger.info(response)
        return response.StatusCode ?? 500
    } catch (error) {

        logger.error("Failed to invoke Lambda", error)
        return 500
    }
}

The last bit of logic adds the implementation of the onKeyDown event handler. We parse the configured settings while adding defaults for anything that's missing and then call the Lambda function. Any return codes in the 200 range are considered good, everything else concerning. Here, showOk will display a ✅ on the key for a few seconds and showAlert will display ⚠️ for a couple of seconds.

// ...
@action({UUID: "com.mauricebrg.lambda-invoke.invoke-async"})
export class InvokeAsync extends SingletonAction<InvokeAsyncSettings> {

    // This function is called when the button is pressed
    public override async onKeyDown(ev: KeyDownEvent<InvokeAsyncSettings>): Promise<void> {

        const settings = parseSettingsAndAddDefaults(await ev.action.getSettings())
        // Now we know that the lambda settings aren't empty
        const statusCode = await invokeLambdaAsync(settings.lambdaSettings!)

        await ev.action.setTitle(`Response\n${statusCode}`)
        if (200 <= statusCode && statusCode < 300) {
            await ev.action.showOk()
        } else {
            await ev.action.showAlert()
        }
    }

}
// ...

This is it. Once you press the button, it will invoke the Lambda function and show the return code of the invoke API. Note that this is not the return code of the Lambda itself as the invocation is asynchronous. There's a long list of things that could be improved in terms of error handling and better feedback to the user, but we're already running long here, so let's finish up. Before we'll bundle up our plugin, let's remove the sample action and its assets. Delete the files from the following list, remove the instantiation + import from src/plugin.ts and remove the action from the manifest.json.

com.mauricebrg.lambda-invoke.sdPlugin/imgs/actions/counter/*
com.mauricebrg.lambda-invoke.sdPlugin/ui/increment-counter.html
src/actions/increment-counter.ts

Next, we check that everything is fine by running the validator on the plugin:

$ streamdeck validate com.mauricebrg.lambda-invoke.sdPlugin
✔ Validation successful

Finally, we packaged it into a distributable plugin:

$ streamdeck pack com.mauricebrg.lambda-invoke.sdPlugin
📦 AWS Lambda Invoke (v0.1.0.0)

Plugin Contents
├─  993 B     manifest.json
├─  1.3 kB    ui/invoke-async.html
├─  6.1 kB    imgs/.DS_Store
├─  1.1 kB    imgs/plugin/category-icon.png
├─  2.4 kB    imgs/plugin/category-icon@2x.png
├─  53.1 kB   imgs/plugin/marketplace.png
├─  123.1 kB  imgs/plugin/marketplace@2x.png
├─  6.1 kB    imgs/actions/.DS_Store
├─  6.1 kB    imgs/actions/lambda/.DS_Store
├─  1.8 kB    imgs/actions/lambda/lambda.svg
├─  20 B      bin/package.json
└─  917.9 kB  bin/plugin.js

Plugin Details
  Name:           AWS Lambda Invoke
  Version:        0.1.0.0
  UUID:           com.mauricebrg.lambda-invoke
  Total files:    12
  Unpacked size:  1.1 MB
  File name:      com.mauricebrg.lambda-invoke.streamDeckPlugin

✔ Successfully packaged plugin

lambda-invoke/com.mauricebrg.lambda-invoke.streamDeckPlugin

You can now distribute the com.mauricebrg.lambda-invoke.streamDeckPlugin file from the directory as your plugin. If you want to publish it on the Elgato Marketplace, you can read more about it in the documentation. You can find the complete code on Github, feel free to extend it or build your own actions. Here are some suggestions what to do next:

Implement a synchronous Lambda invocation
Add a linter, prettier and JSDoc strings as explained in the docs

I hope you enjoyed this post and learned something new.

— Maurice

The thing I keep relearning about the S3-SNS/SQS integration

Maurice Borgmeier — Tue, 10 Jun 2025 13:31:50 +0000

Occasionally, you'll do something that you think you've done dozens of times before and are then surprised it no longer works. While setting up a log delivery mechanism for Splunk, I had one of these experiences again. (Feel free to replace relearning with forgetting in the headline.)

Splunk's preferred method of ingesting log data from AWS is the SQS-based S3 input. In a nutshell, you ensure that all logs end up in an S3 bucket. That bucket is configured to send all object create events to an SNS topic (so that multiple systems can subscribe), to which an SQS queue is subscribed. Splunk subsequently consumes the object create events from the queue and ingests the corresponding objects from S3.

That's straightforward—until you add encryption to the mix. Per the customer's policy, queues and topics need to be KMS-encrypted. Having configured and deployed it that way in Terraform, I was surprised that nothing was showing up in Splunk. I knew that AWS does a bunch of validation when you create an event subscription in S3 or an SQS subscription in SNS. One thing AWS doesn't validate is if S3 has permission to send encrypted messages to SQS -i.e., if S3 is allowed to call kms:Encrypt and kms:GenerateDataKey on the key. If that's missing, you end up in a situation where all configuration seems fine, without anything being delivered.

Eventually, I remembered that I had a similar issue years ago, and all I needed to do was allow the S3 service principal to call the aforementioned KMS APIs. At least I'm not the only one who seems to be running into this, as this topic is part of the troubleshooting steps on re:Post. After adding the service principal, everything ran smoothly and logs showed up in Splunk.

This post is just me trying to make sure I don't have to relearn this again. Once I write things down, it tends to stick.

Which AWS Regions are Lambda@Edge functions executed in?

Maurice Borgmeier — Thu, 06 Feb 2025 13:25:18 +0000

Lambda@Edge is one of two ways to run custom code as part of the request flow in CloudFront. In contrast to typical Lambda functions, they must be deployed to us-east-1 (North Virginia), and CloudFront will schedule them in regions across the world close to your users to optimize the response latency. Lambda@Edge functions are handy if you want to run custom logic inside the request or response path that doesn't fit within the constraints of what's possible with CloudFront functions.

Unfortunately, at the time of writing this, the AWS documentation doesn't mention in which regions functions can be executed. It just explains that replicas of the function are created in regions around the world. The only two sources I could find on the topic are this Reddit post by a random person and an AWS blog post that briefly mentions they're running in Regional Edge Caches.

Why would you even care? Isn't this an implementation detail? In some ways, it shouldn't matter much, and the only reason that I care about this is logging. When Lambda@Edge functions are executed, they write their logs in a log stream in the region where the code is running. That means you'll find log groups called /aws/lambda/us-east-1.{function-name} scattered around the world with no central visibility. The aforementioned blog post details how to use a firehose stream to aggregate the logs into an S3 bucket for further analysis.

To do that, we need to deploy log groups in all regions that have these Regional Edge Caches, and the 2019 post mentions 11 of them. On the CloudFront features page, we get a nice overview of CloudFront locations around the world. We care about the ones with a red dot.

The website also offers a list view that's a little more useful than the pretty map since it has actual names. This contains nice human-readable names. Unfortunately, they don't mention the API code (e.g., us-east-1), which you'd actually need to pre-create log groups.

Since I'm a big fan of spending too much time automating things that should really be done precisely once manually, I figured out how to create a list of region codes in which Lambda@Edge functions can be scheduled. First, I used the developer tools to figure out which endpoints the website uses to get all these CloudFront locations. AWS commonly uses a central API to retrieve content listings for its website, that's available at https://aws.amazon.com/api/dirs/items/search.

This API is undocumented, so you probably shouldn't rely on it, but it has been around for a long time. The website can retrieve all kinds of information from here, and if we specify the correct directory ID (which we get by finding the correct request in the network log), we get a list of the CloudFront points of presence that are visible on the maps. The data structures it returns look something like this (just a single list item here):

{
  "item": {
    "id": "cf-map-pins#namer-us-zelienople",
    "locale": "en_US",
    "directoryId": "cf-map-pins",
    "name": "North America United States Zelienople",
    "dateCreated": "2024-02-15T18:28:10+0000",
    "dateUpdated": "2024-03-22T18:51:47+0000",
    "additionalFields": {
      "x": "64.4",
      "pinName": "Zelienople",
      "y": "41",
      "pinDescription": "United States"
    }
  },
  "tags": [
    {
      "id": "GLOBAL#location#namer",
      "locale": "en_US",
      "tagNamespaceId": "GLOBAL#location",
      "name": "North America",
      "description": "North America",
      "dateCreated": "2020-02-03T05:29:03+0000",
      "dateUpdated": "2022-02-03T03:31:34+0000"
    },
    {
      "id": "cf-map-pins#map-format#embedded-pops",
      "locale": "en_US",
      "tagNamespaceId": "cf-map-pins#map-format",
      "name": "Embedded POPs",
      "description": "<p>Embedded POPs</p>\n",
      "dateCreated": "2024-02-15T19:34:19+0000",
      "dateUpdated": "2024-02-15T19:34:19+0000"
    },
    {
      "id": "GLOBAL#infrastructure-type#region",
      "locale": "en_US",
      "tagNamespaceId": "GLOBAL#infrastructure-type",
      "name": "Region",
      "description": "Region",
      "dateCreated": "2021-05-19T07:48:42+0000",
      "dateUpdated": "2022-02-03T03:31:20+0000"
    }
  ]
}

It would be nice if the data structure included any API names, but sadly that's not available from this web service. Time to start scraping. I wrote a small python script that queries the data from this endpoint and filters it to the regional edge caches. Regional edge caches can be identified by having a specific tag id attached.

import logging
import sys

import requests

LOGGER = logging.getLogger(__name__)
LOGGER.addHandler(logging.StreamHandler(sys.stdout))
LOGGER.setLevel(logging.DEBUG)


def fetch_cloudfront_pops() -> list[dict]:

    more_content = True
    page = 0

    all_items = []

    while more_content:
        more_content = False

        response = requests.get(
            "https://aws.amazon.com/api/dirs/items/search",
            params={
                "item.directoryId": "cf-map-pins",
                "sort_by": "item.additionalFields.y",
                "sort_order": "desc",
                "size": "500",
                "item.locale": "en_US",
                "page": page,
            },
            timeout=10,
        )

        for item_container in response.json()["items"]:
            more_content = True

            item = item_container["item"]
            item["tags"] = item_container["tags"]

            all_items.append(item)

        page += 1

    return all_items


def is_regional_edge_pop(item: dict) -> bool:

    for tag in item["tags"]:
        if tag["id"].endswith("#regional-edge-caches"):
            return True
    return False

Running this gives us a list of regional edge caches and the country they're located in:

if __name__ == "__main__":
    all_pops = fetch_cloudfront_pops()
    regional_pops = [pop for pop in all_pops if is_regional_edge_pop(pop)]

    regional_pop_names_and_descriptions = [
        (x["additionalFields"]["pinName"], x["additionalFields"]["pinDescription"])
        for x in regional_pops
    ]
    LOGGER.info("Regional Edge Caches: %s", regional_pop_names_and_descriptions)

Regional Edge Caches: [('Seoul', 'South Korea'), ('Tokyo', 'Japan'), ('Sao Paulo', 'Brazil'), ('Dublin', 'Ireland'), ('Oregon', 'USA'), ('Mumbai', 'India'), ('Ohio', 'USA'), ('Sydney', 'Australia'), ('London', 'UK'), ('Frankfurt', 'Germany'), ('Northern Virginia', 'USA'), ('California', 'USA'), ('Singapore', 'Singapore')]

Now, we just need to map these to the correct API codes, which should be easy, right? We just need to find an API endpoint that maps the human-readable names of AWS regions to their API code. The official APIs allow you to get a list of API codes through the account.ListRegions API, which is insufficient because it doesn't contain the human-readable names.

After some digging, I found the AWS website that allows you to check the regional availability of services. The developer tools showed that it requested a locations.json which happens to contain exactly the data I was interested in.

// Excerpt
{
  "Africa (Cape Town)": {
    "name": "Africa (Cape Town)",
    "code": "af-south-1",
    "type": "AWS Region",
    "label": "Africa (Cape Town)",
    "continent": "Africa"
  },
  // ...
}

The last problem to solve was mapping the display names from the console to the names in the CloudFront diagram. CloudFront sometimes uses names that are very close to the console display name, e.g., "Cape Town" and "Africa (Cape Town)", but also variations such as "Northern Virginia" instead of "US East (N. Virginia)", which make matching a bit more annoying. The biggest confusion is the Ireland/Dublin region. CloudFront refers to it as "Dublin," and the console display name is "EU West (Ireland)".

I ended up solving this with difflib.get_close_matches from the Python standard library. It accepts a word and a list of keywords and returns the n closest matches from the keyword list using a similarity score. This allowed me to find the best match in the console display names based on the city's name. Since we have to deal with Dublin/Ireland, I had to check based on the description (country name) if the city yielded no result.

def get_region_info() -> dict[str, dict]:
    global_infrastructure = requests.get(
        "https://b0.p.awsstatic.com/locations/1.0/aws/current/locations.json",
        timeout=10,
    ).json()

    return {
        key: value
        for key, value in global_infrastructure.items()
        if value["type"] == "AWS Region"
    }

if __name__ == "__main__":
    # ...

    region_info = get_region_info()

    for regional_pop_name, description in regional_pop_names_and_descriptions:
        matches = difflib.get_close_matches(
            regional_pop_name, region_info.keys(), n=1, cutoff=0.4
        )

        if not matches:
            LOGGER.debug(
                "Failed to match %s by name, trying description (%s)",
                regional_pop_name,
                description,
            )
            matches = difflib.get_close_matches(
                description, region_info.keys(), n=1, cutoff=0.3
            )
            if not matches:
                raise RuntimeError(f"Unable to map {regional_pop_name}")

        region_name = matches[0]
        api_name = region_info[region_name]["code"]
        LOGGER.info(
            "The CF-Name %s most closely matches %s (%s)",
            regional_pop_name,
            region_name,
            api_name,
        )

Running my code showed that the number of regional edge caches has increased from 11 in 2019 to 13 in early 2025 and are as follows:

Region Name	Region Code
Asia Pacific (Mumbai)	ap-south-1
Asia Pacific (Seoul)	ap-northeast-2
Asia Pacific (Singapore)	ap-southeast-1
Asia Pacific (Sydney)	ap-southeast-2
Asia Pacific (Tokyo)	ap-northeast-1
EU (Frankfurt)	eu-central-1
EU (Ireland)	eu-west-1
EU (London)	eu-west-2
South America (Sao Paulo)	sa-east-1
US East (N. Virginia)	us-east-1
US East (Ohio)	us-east-2
US West (N. California)	us-west-1
US West (Oregon)	us-west-2

Conclusion

Reddit was correct all along. This information shouldn't be hard to find - I'd expect this list or maybe a table to exist somewhere in the CloudFront docs. Ideally, there should be an official API endpoint so you can automate the process more quickly. Granted, the list doesn't seem to change much, but AWS clearly has that information available and could make our lives easier.

If you liked this post, you might also like the one telling the story of how I spent a few hours using advanced technology to save $2.

The script is available on Github.

— Maurice

Integrating HubSpot with AWS Lambda

Maurice Borgmeier — Fri, 06 Dec 2024 12:32:30 +0000

Like many organizations, tecRacer uses HubSpot as a CRM. Integrating Hubspot with other (internal) systems enables smooth workflows for everyone involved. Since I recently built a custom integration, I thought it may be helpful to explain how to set up a secure interface with AWS.

In our use case, we wanted to react to changes inside HubSpot and update data in another system. This meant we were looking for a way to have webhook-like capabilities, allowing us to use Lambda to run custom logic.

Initially, we explored Zapier as an integration platform, which would have enabled us to trigger a Lambda function in response to a HubSpot event. While that would have worked, the drawback is that the target system doesn't have a Zapier integration, which means that Zapier would have just been an expensive trigger system. If source and destination have Zapier integrations, definitely check it out, though. For us, DIY was the best approach.

To build a custom integration, we need to create a private app in HubSpot, which these docs outline. The private app has two main functions:

Enable the use of the HubSpot API to read and write data
Configuration of event-driven integrations through webhooks

On the AWS side of the picture, we want to keep things as simple as possible. The AWS part is primarily responsible for accepting the webhook trigger and then communicating with HubSpot and our other system. A Lambda function URL is the easiest way to provide an endpoint for the webhook. An API Gateway in front of Lambda would have also worked, but we don't need custom domains or any of the other things it supports. This Lambda function can then do whatever it wants with the data.

The main reason why I wrote this post is authentication. Lambda function URLs support IAM authentication, but that's not supported by HubSpot. Instead, HubSpot cryptographically signs the events it sends and we can verify that signature to check that the request is coming from our private app. Signature verification requires access to the app's client secret. Later communication with the HubSpot API needs the app's access token, both of which we're storing in the AWS Secrets Manager.

Based on this, the implementation of the Lambda function looks roughly like this:

def lambda_handler(event, _context):
    access_token, client_secret = get_access_token_and_client_secret()

    if not is_hsv3_signature_valid(event, client_secret):
        return {"statusCode": "403"}

    api_client = HubSpot(access_token=access_token)
    # ...

HubSpot provides the signature (X-HubSpot-Signature-v3) as well as the timestamp used to create it in HTTP headers that are sent to the Lambda function URL. There are multiple versions of this signature; the current recommendation is to use v3, which I will talk about here.

Before we even compute the signature, the documentation recommends that we reject any request where the signature timestamp (X-HubSpot-Request-Timestamp) is older than five minutes, presumably as a protection against replay attacks. Assuming our request is within that five-minute window, we can proceed to compute the signature. For that, we need some information from the request, i.e., the event data structure that invokes our function URL:

request_method from event["requestContext"]["http"]["method"]
request_uri, which is concatenated from event["requestContext"]["domainName"] and event["requestContext"]["http"]["path"]
request_body from event["body"]
timestamp from event["headers"]["x-hubspot-request-timestamp"]

Next, we concatenate the values in the order that I listed them and run them through an HMAC SHA-256 function, with the key being the private app's client secret. The result of this is then encoded using Base64, which ends up being our signature. If our computed signature matches the one in X-HubSpot-Signature-v3, we can be sure that the request originates from our HubSpot app and wasn't modified in transit.

The Python implementation requires no external dependencies as all components involved are part of the standard library:

import base64
import hashlib
import hmac

def is_hsv3_signature_valid(
    event: dict, hs_client_secret: str, recent_timestamps_only=True
) -> bool:
    """
    Validates the signature on an Event received by a Lambda Function URL sent
    by hubspot according to the v3 Signature spec.

    https://developers.hubspot.com/beta-docs/guides/apps/authentication/validating-requests

    Parameters
    ----------
    event : dict
        The event the lambda function receives.
    hs_client_secret : str
        The client secret of the (private) app.
    recent_timestamps_only : bool, optional
        Enable or disable age verification on the timestamp, by default True

    Returns
    -------
    bool
        True if the signature is valid, otherwise false.
    """

    five_minutes_ago_epoch_ms = (
        datetime.now() - timedelta(minutes=5)
    ).timestamp() * 1000

    request_method = event["requestContext"]["http"]["method"]
    request_uri = f"https://{event['requestContext']['domainName']}{event['requestContext']['http']['path']}"
    request_body = event["body"]
    timestamp = event["headers"]["x-hubspot-request-timestamp"]

    if int(timestamp) < five_minutes_ago_epoch_ms and recent_timestamps_only:
        LOGGER.warning(
            "Timestamp too old, must be within the past 5 minutes! %s should be > %s",
            timestamp,
            five_minutes_ago_epoch_ms,
        )
        return False

    hmac_payload = request_method + request_uri + request_body + timestamp
    sha256_hmac = hmac.new(
        hs_client_secret.encode("utf-8"),
        msg=hmac_payload.encode("utf-8"),
        digestmod=hashlib.sha256,
    )

    expected_signature = base64.b64encode(sha256_hmac.digest()).decode("utf-8")
    actual_signature = event["headers"]["x-hubspot-signature-v3"]

    return expected_signature == actual_signature

I chose to add the recent_timestamps_only parameter to make the five-minute time window optional, which makes testing this a lot easier. Outside of tests, you should only deactivate it if your system's clock is very unreliable. The docs also mention the need to decode URL parameters, which I skipped here because the webhook calls the root path without any parameters. If you want to add parameters, you may want to look into urllib.parse.parse_qs.

This implementation effectively ensures that only authentic data will be further processed. However, one drawback of function URLs is that anyone could call the function URL, which may lead to economic attack vectors, but given that the URL is unpredictable and the time it takes for the validation to detect invalid signatures (or fail) is very short, the risk is limited.

I hope this helps some people who are currently trying to validate these kinds of requests.

— Maurice

How I spent a few hours using advanced technology to save $2

Maurice Borgmeier — Fri, 01 Nov 2024 08:33:36 +0000

I usually rely on Infrastructure as Code to build my solutions - usually, not always. In my AWS Playground account that I use for demos and exploration, there are some resources that I've created manually over the years and forgot to delete.

I check Cost Explorer at least once per month to identify resources that could be turned off, and this time, I decided to take a closer look at my KMS charges. They're nowhere near exorbitant at slightly more than $4 per month, but given that barely anything is running in this account, $4 ends up being a significant fraction of the bill.

At this point, I should probably point out that there are opportunity costs involved when I spend time optimizing a monthly $4 charge. If it was only about that, this probably wouldn't break even in quite some time. Since I use this as an opportunity to educate myself (and potentially you) about approaches to tackle these kinds of issues, which expands my skills for real projects, I'm willing to spend a bit more time on that.

The lion's share of the $ ~4 / month is the $1 per month that AWS charges per Customer Managed Key and since I have precisely four of those, this accounts for the costs aside from a couple cents for API calls. Now the question is: what are these keys being used for? Taking a look at the console reveals only Key IDs and Aliases, although individual keys can also have a description. (This screenshot is post-cleanup; I forgot to take one in advance)

While there's a description field, whoever created these (me) never bothered to add any text. Fortunately, two had aliases defined, which allowed me to identify the stack they belonged to, and I was able to schedule one of them for deletion (as well as clean up a few other resources). The other key with an alias is still being used, and this leaves us with two unaliased keys. Sometimes, the resource policy on the key or any tags they might have attached can help us figure out what their purpose is, but in my case, none of them were insightful.

This leaves me with two questions to answer before I can consider deleting the keys:

Are these keys actively used?
Are there resources where the keys are part of the configuration?

To answer my questions, I considered a few data sources and tools:

AWS Config as the central repository to track changes to my resources
AWS CloudTrail, which provides an audit trail of all, well most, API calls
IAM to identify references to these specific keys in policies
Steampipe, which can be used to query resources using SQL

Additionally, I considered the time-honored scream test, where you just shut things off and wait for someone to scream. However, this being my own playground environment with less-than-ideal monitoring led me to discard this option. If you're the only potential screamer, it's less fun.

AWS Config

Searching for the key by its ID in AWS Config, led me to the resource timeline for the key, which in turn has an integration with CloudTrail.

I could see that there had been no config changes to the key in a while and that there's some kind of API activity that CloudTrail reports on. Unfortunately, there was so much noise where the IAM Access Analyzer (not useful for this question) and AWS Config itself were accessing the key that this wasn't very useful. A view of the connected resources, i.e., those that reference the current resource, would have been handy here, but there is none.

Fortunately, there is the advanced query feature that allows you to select resources using SQL. For the data schema, the console just features a link to Github where you can learn a bit about the attributes and data types. Some visual explorer would have been nice here. Since I wasn't too excited about looking through literally hundreds of JSON files to get the schema, I asked Claude 3.5 to help.

Naturally, this failed because, for some obscure reason, the like syntax doesn't support a wildcard character at the beginning of the query.

Wildcard matches beginning with a wildcard character or having character sequences of less than length 3 are unsupported

When I prodded Claude more about that, it spit out SQLs with many OR conditions for all possible parameter names and filters to resources I didn't ask for. I was a bit annoyed; I wanted it to treat all configuration options as strings and just find an ARN in there (Amazon Q wasn't helpful here, either). Looking at the documentation for advanced queries leads me to believe that my use case is clearly too advanced and, thus, not supported. Is searching for connections between resources such an obscure requirement?

AWS CloudTrail

Disappointed by AWS Config, I decided to check CloudTrail. I was confident that this would at least allow me to figure out if something was actively using the key. Since the in-console event history is limited to 90 days of management events, I created an Athena table that allowed me to query the raw data in S3 since the beginning of time - a bit more than five years when this account was created.

SQL with nested data structures is always a bit finicky, so I tried to give Claude another chance to redeem itself and was (predictably) disappointed. Some good old-fashioned googling about properly unnesting arrays yielded the following query, which lists the API calls where the KMS Key in question is the target, which user agent performed them, how many times, and when the last call happened.

SELECT eventsource
     , eventname
     , count(*) as n_requests
     , max(eventtime) as latest_request
     , useragent
  FROM my_athena_table,
UNNEST (resources) as t(resource)
 WHERE 1=1
   AND resource.arn = 'arn:aws:kms:eu-central-1:<account>:key/<key-id>'
 GROUP BY eventsource, eventname, useragent
 ORDER BY eventsource, eventname, useragent

The query took some time to run because Athena isn't optimized to work with thousands of compressed tiny json files, but after a bit more than two minutes, I got my result. That's how I found out that earlier this year, I had been using the key for some Textract encryption experiments, and since then, there had only been Describe* operations. This means it was most likely fine to delete the key. Unfortunately, CloudTrail couldn't really help me figure out where the key is currently configured, as not all parameters and responses are fully captured (size constraints).

AWS IAM / Steampipe

Last, but not least, I'm going to try using Steampipe to query AWS resources using SQL. Steampipe makes AWS resources available as tables, and I could use the following statements to find out which customer-managed policies and inline policies reference the key.

-- Find customer-managed policies that reference the key
select *
  from aws_iam_policy as p
 where p.policy::text like '%arn:aws:kms:eu-central-1:<account>:key/<key-id>%'

-- Find users with inline policies that reference the key
select *
  from aws_iam_user 
where inline_policies::text like '%arn:aws:kms:eu-central-1:<account>:key/<key-id>%'

-- Find roles with inline policies that reference the key
select *
  from aws_iam_role
where inline_policies::text like '%arn:aws:kms:eu-central-1:<account>:key/<key-id>%'

This is how I found out that I may have created the key using terraform, because a few roles showed up that had clearly been create through terraform, but none of those things are still in use.

Unfortunately, you can't really use Steampipe to figure out all the places that reference the key in a practical manner, as it relies on API calls to AWS, and you'd be querying every AWS service. The service that already has all of this information in one spot is AWS Config. Unfortunately, its SQL feature is severely limited. It's possible to export an AWS Config snapshot to S3 and query that using Athena if you want to jump through a couple more hoops, but this is where I decided that I had enough info to delete the KMS Key.

In about a week I'm going to have only two keys - the one that's still in use and another one for Demos, this time with a description and tags.

What can we do to avoid getting in this position in the first place? The first step is tagging your resources. Not all resources support it, but most do. You can start by adding an application tag, allowing you to view the resources as a group in the resource explorer. Also, Infrastructure as Code will usually help you. The tools make it easy to apply tags to all resources (you still have to configure it, though), and this gives you a much better insight into what's going on.

In my case, part of the resources were created through IaC, so clearly, this is not a silver bullet; you still have to do it properly. On a related note - if someone has seen my Terraform state and code for that experiment, please let me know.

Conclusion

In conclusion, we learned about a few tools that we can leverage to identify who or what is using or referencing our resources, which may help with cost optimization or at least allocation. Since this process is a bit annoying after the fact, the old saying "an ounce of prevention is worth a pound of cure" holds true.

— Maurice

I considered calling this blog "How I slashed my KMS bill by 50% "but decided against it because, technically, it's a bit less than 50%.

Building Data Aggregation Pipelines using Apache Airflow and Athena

Maurice Borgmeier — Mon, 23 Sep 2024 09:27:45 +0000

Decisions about running a company are rarely made based on individual transactions. Instead, business intelligence is usually derived from aggregate data, e.g., daily sales by region/product/demographic. Individual transactions are frequently aggregated in advance to analyze this data in a timely manner and make slicing and dicing it a pleasant experience. In this blog post, we'll discuss building this kind of aggregation pipeline with Airflow and Athena.

We'll base this post on the TPC-H dataset, which is commonly used to benchmark data warehouses and describes a typical e-commerce application. I explained how to provision this dataset into your datalake in the past: Making the TPC-H dataset available in Athena using Airflow, so go check that out if you haven't already. I'll also assume that you're familiar with both Athena and Airflow.

The TPC-H dataset features the orders table, which stores individual transactions, the total order volume, and references to some dimensions, e.g., customer info. We can use these to determine the market segment, country, and region in which the sale occurred by joining the orders table with other dimensions. As an example, we can use the following query to compute the sales volume per market segment, nation, and region for the 1st of August 1998:

/*
Daily sales per market segment, nation, and region
*/
select c_mktsegment,
    n_name,
    n_nationkey,
    r_name,
    r_regionkey,
    o_orderdate,
    sum(o_totalprice) as total_revenue,
    year(o_orderdate) as "year",
    month(o_orderdate) as "month",
    day_of_month(o_orderdate) as "day_of_month",
    day_of_week(o_orderdate) as "day_of_week"
from customer c
    join orders o on c.c_custkey = o.o_custkey
    join nation n on c_nationkey = n.n_nationkey
    join region r on r_regionkey = n.n_regionkey
where o_orderdate = date('1998-08-01')
group by c_mktsegment,
    n_name,
    n_nationkey,
    r_name,
    r_regionkey,
    o_orderdate
    ;

As mentioned, this aggregates the day's transactions by market segment, nation, and region, which reduces the number of records our BI tool, e.g., Quicksight, has to process when creating interactive visualizations by several orders of magnitude.

Before we turn this individual statement into a data pipeline in form of an Airflow Directed Acyclic Graph (DAG), let's talk about some qualities we like to see in our data pipelines or ETL Jobs. Well-designed ETL jobs should include some form of error handling that tries to resolve commonly occurring problems on its own.

Additionally, it should support reprocessing existing data because errors may be discovered and corrected later, and these fixes should be incorporated into the output. Visibility into a running process and the overall status of our pipeline over time should also be built in.

We will tackle this using Airflow for orchestration and Athena to perform the actual data processing. The code I'm going to show now is available in the companion repo on Github. With regard to errors, the most common remediation step is to retry the operation, which Airflow can do for us, so it's just a matter of configuration.

with DAG(
    dag_id="daily_sales_aggregation",
    dag_display_name="Daily Sales Aggregation",
    default_args={"retries": 1, "retry_delay": timedelta(minutes=5)},
    # ...
) as dag:
    # ...

The more specific error condition we need to consider is that there isn't a table when writing data for the first time. Since I like my DAGs to be self-contained, I will have the DAG respond to this error by creating the table and performing the insert operation again. Unfortunately, that's not straightforward. First, we add a failure callback, which serializes the error messages and stores it as an XCom, basically a key-value store that allows us to fetch the message later from another task.

def _add_exception_to_xcom(context):
    context["task_instance"].xcom_push("insert_error", str(context["exception"]))

insert_into_aggregate_table = AthenaOperator(
    task_id="insert_into_aggregate_table",
    task_display_name="Insert into Daily Sales",
    on_failure_callback=_add_exception_to_xcom,
    **insert_params,
)

Then, a branch operator retrieves that error message and checks if it's the expected Table not found message. In that case, we move on to the create_aggregate_table task. Otherwise, we raise an error because that could indicate some other unknown issue.

@task.branch(
    task_id="handle_result",
    task_display_name="Table doesn't exist?",
    trigger_rule="all_failed",
)
def is_table_missing(**context):

    error_message = context["task_instance"].xcom_pull(
        key="insert_error", task_ids="insert_into_aggregate_table"
    )

    if fnmatch.fnmatch(error_message, "*Error: Table * not found in database*"):
        LOGGER.info("Table doesn't exist, next step: create table.")
        return "create_aggregate_table"

    raise AirflowException(f"Unknown error during insert {error_message}")


handle_missing_table = is_table_missing()
insert_into_aggregate_table >> handle_missing_table

Of course, we could forego this complexity and ensure we create the table manually before we run the process for the first time. I'm not a huge fan of manual tasks, though, and this way, the DAG takes care of the whole table's lifecycle, aside from deleting it.

The other quality we were looking for is the ability to reprocess data, which means that we want to achieve a mild form of idempotency, i.e., the ability to re-run the pipeline with the same inputs and underlying data and get the same output. Given that we're in a data lake situation and Athena can't execute delete statements, we need to remove any underlying data before we insert data. Otherwise, we may end up with duplicate records when data is reprocessed.

For this to work, we need to ensure that each insert statement creates a new partition, i.e., we're going to partition our daily_sales table by year, month, and day. This will, in turn, result in the data for each partition being stored in a predictable S3 location and allows us to use the S3DeleteObjectsOperator to clean any objects from that prefix before trying to insert data. If there is no pre-existing data, this costs us one ListObjectsV2 API call per day, which is very cheap.

clean_s3_prefix = S3DeleteObjectsOperator(
    task_id="clean_s3_prefix",
    task_display_name="Delete Existing data from S3",
    bucket="{{params.s3_bucket}}",
    prefix="{{params.s3_prefix}}{{params.daily_sales_table_name}}/{{ macros.ds_format(ds, '%Y-%m-%d', 'year=%Y/month=%-m/day_of_month=%-d/') }}",
)

Assembling these building blocks into a DAG gives us the following pipeline. First, we delete data for the daily partition if there is any. Next, we try to insert data into the table. If the table doesn't exist, we create it and repeat the insert.

Aside from the initial run, only the first two tasks should ever be triggered, and usually, the first one won't do much. If we ignored all repeatability and error handling, we could just have one insert task. Making something production-ready and maintainable can add some complexity, which you won't have if you only consider the happy path.

Something to consider with the TPC-H dataset is that the data covers the years from 1992 to 1998, so without further modifications we can't really simulate newly incoming data. Instead, we can set a logical date when we run the DAG to run it for a historical date. You'll find the time frame hard-coded in the DAG parameters on GitHub.

To create our table, we can set the catchup parameter to true, which will cause it to compute the daily sales for the whole time period. Since that may take a while, I've also created a second DAG that does monthly aggregation, where the catchup won't take as long.

We can also use the airflow dags backfill command in order to trigger or re-run DAG runs for a specific time period. The following command triggers DAG runs for November 1994. By default, it won't re-run for dates that it has already processed, which we could change by adding the --reset-dagruns parameter.

$ airflow dags backfill daily_sales_aggregation \
--start-date 1994-11-01 \
--end-date 1994-11-30

The calendar view lets us view the progress of our backfill operation.

I should mention one other detail here. I've made life simple for myself by using the {{ds}} template variable for the day filter. This means Airflow will filter the data based on the day the DAG is running on (unless you overwrite the logical date). Depending on when your data is updated, you may have to do some date arithmetic here. If, for example, you have to process the data of the previous day, you'd have to replace {{ds}} with {{ macros.ds_add(ds, -1) }}. There are more considerations with regard to data consistency and scheduling, but this is already getting long, so I'll talk about it another day.

In this post, I've shared two DAGs with you that can be used to create daily or monthly data aggregation pipelines. I've also outlined some features of a production-ready data pipeline, which, on the surface, complicate things but make the pipeline more robust in the real world. I hope you learned something new.

We're happy to help you set up your data pipelines and BI solutions. Go check out our offerings in the data analytics and machine learning space!

— Maurice

Special thanks to my friend Peter for sharing his expertise and feedback!

Cover Photo by Isaac Smith on Unsplash (yes, I chose it because of the axes)

How to accidentally create read-only DynamoDB items

Maurice Borgmeier — Sat, 14 Sep 2024 06:58:19 +0000

In a recent Developing on AWS training, I taught my students the basics of DynamoDB. I usually do that by building up the concepts on a digital whiteboard. I think this helps students connect the many pieces of information to form a mental model of the service. Additionally, it's more engaging for me as an instructor, invites conversations, and is a welcome break from the usual PowerPoint slides.

Having used DynamoDB for a couple of years and delivered that training many times, I rarely get questions about the service that I can't answer off the top of my head. This time, there was a new one that intrigued me. A student asked me what happens if you add an item to a table with a global secondary index, where the attribute name is one of the index attributes, but the data type differs. I was under the impression that the datatypes are only enforced for the key attributes of the base table. Consequently, I thought the item would not appear in the global secondary index.

Turns out I was wrong. Let's find out what actually happens.

First, we define a table with a simple primary key, PK, and the data type string. Later, we'll add a global secondary index with the attribute GSI1PK of type string as its partition key.

Before we create the global secondary index, we add a few items to the table:

// 1: Item with correct datatypes
{"PK": {"S": "ok"},"GSI1PK": {"S": "ok"}}
// 2: Item with incorrect datatype in the base table
{"PK": {"N": "123"},"GSI1PK": {"S": "ok"}}
// 3 & 4: Items with incorrect datatypes in the non-existent GSI
{"PK": {"S": "ok2"},"GSI1PK": {"N": "123"}}
{"PK": {"S": "ok3"},"GSI1PK": {"N": "123"}}

Writing all but the second item succeeds. When attempting to store the item with the Number data for the PK attribute, we receive a ValidationException with the message:

One or more parameter values were invalid: Type mismatch for key PK expected: S actual: N

This is what I expected—it enforces the data types for the base table's index. At this point, it doesn't care about the data type for the GSI1PK attribute as it doesn't have any special meaning yet, so it accepts the PutItem calls for items 3 and 4.

Now, I add a global secondary index for the GSI1PK attribute. The nice thing about GSIs is that they can be added at any point in time, which is one of the things that makes them more flexible than their counterpart, the local secondary index. After waiting for it to backfill and become active, i.e., propagate the already existing data into the index, we can verify that only the item with the valid data types is available in this index by scanning it.

Items in GSI1: [{'PK': {'S': 'ok'}, 'GSI1PK': {'S': 'ok'}}]
Only the item with the correct datatypes ends up in GSI1

So far, this matches what I expected. The index skips items that have the key attributes but incorrect data types. My expectations weren't met when I tried to store another item, like the ones in 3 and 4 in the index. This time, I got a familiar error:

(ValidationException) One or more parameter values were invalid: Type mismatch for Index Key GSI1PK Expected: S Actual: N IndexName: GSI1

This means as soon as a secondary index is created, DynamoDB starts enforcing the data type for all items written after it has been created. We could stop here, but this isn't the full story. I was wondering what happens to the items 3 & 4, those with the incorrect data types in what has now become an index attribute.

I tried updating one of the items using the UpdateItem API to add another index-unrelated attribute to it and was met with the following error:

(ValidationException) The update expression attempted to update the secondary index key to unsupported type

Interesting. It effectively prevents us from changing the item. So, are we now stuck with a read-only item? Not quite. It seems DeleteItem is unaffected, and we're able to clean up the mess. But what if you care about the data and prefer to try to fix it? You're in luck - there's one UpdateItem operation that DynamoDB will allow you to do:

client.update_item(
    Key={"PK": {"S": "ok2"}},
    UpdateExpression="SET GSI1PK = :val",
    ExpressionAttributeValues={":val": {"S": "123"}},
    TableName=TABLE_NAME,
)

You can replace the existing numeric value of GSI1PK with a string attribute, and that will make it writable once again and also add it to the GSI. If you want to try this yourself, I put the script and output on Github.

Having done my experiments, I researched the documentation, and unsurprisingly, I'm not the first to discover this. In detecting and correcting Index Key Violations, AWS outlines what my experiments also show:

"If an index key violation occurs, the backfill phase continues without interruption. However, any violating items are not included in the index. After the backfill phase completes, all writes to items that violate the new index's key schema will be rejected."

They even built a tool to detect and correct these cases, the DynamoDB Online Index Violation Detector tool, which is available on Github. Although it was last updated ten years ago and archived in 2020, it may no longer function as expected, but I haven't tried it.

One of the reasons why I enjoy delivering courses is such questions. Explaining concepts forces you to challenge your own understanding, and doing it in front of others allows you to have them do the same based on their understanding. Everyone benefits, which I find a rewarding experience.

If you think these kinds of courses are right for you, check out our course catalog and get in touch. I'm sure we have a training that will intrigue you.

— Maurice

Making the TPC-H dataset available in Athena using Airflow

Maurice Borgmeier — Thu, 29 Aug 2024 12:47:05 +0000

The TPC-H dataset is commonly used to benchmark data warehouses or, more generally, decision support systems. It describes a typical e-commerce workload and includes benchmark queries to enable performance comparison between different data warehouses. I think the dataset is also useful to teach building different kinds of ETL or analytics workflows, so I decided to explore ways of making it available in Amazon Athena.

Typically, you'd download a data generator from the Transaction Processing Performance Council's (TPC) website and tell it how much data to generate. Afterward, you grab a coffee and another coffee, and at some point, you have the data, which can be imported into your favorite data warehouse.

The waiting part is the one I'm not too excited about, so I decided to explore alternative options. Googling led me to this AWS Github repository that the Redshift team seems to use to run benchmarks on their data warehouse. Conveniently, they've generated datasets already and are just loading them from a public S3 bucket. Very interesting.

We can find datasets ranging from 10GB, over 100GB to 3TB, and even 30TB, which should be plenty for our experiments and learning about data analytics. Conveniently, the repository also includes ddl.sql files for each size that describe both the structure of the database tables as well as their location and format in S3.

Here's an abbreviated example of one such ddl.sql:

/* ... */
create table nation (
  n_nationkey int4 not null,
  n_name char(25) not null ,
  n_regionkey int4 not null,
  n_comment varchar(152) not null,
  Primary Key(N_NATIONKEY)                                
) distkey(n_nationkey) sortkey(n_nationkey) ;
/* ... */
copy nation from 's3://redshift-downloads/TPC-H/2.18/30TB/nation/' iam_role default delimiter '|' region 'us-east-1';
/* ... */

This makes our task significantly easier. For the most part, the plan is now to make this data available as an Athena table in the raw format and then convert it to an optimized format, i.e., (partitioned) parquet using a CTAS (Create Table As Select) statement, leaving Athena to do the heavy lifting.

Naturally, I started with the 10GB dataset because I like quick feedback cycles, and waiting for 3TB transformations didn't seem very desirable. Also, the data is located in us-east-1, and I'm running my things in eu-central-1, which means I (more specifically, my boss) will be paying for data transfer costs on top of the Athena processing fees.

I quickly noticed that the 10 GB dataset is stored in an odd way. If we look at an excerpt from its ddl.sql, we can see that they're copying the data into Redshift from individual files that are located at the same level in S3.

copy region from 's3://redshift-downloads/TPC-H/2.18/10GB/region.tbl' iam_role default delimiter '|' region 'us-east-1';
copy nation from 's3://redshift-downloads/TPC-H/2.18/10GB/nation.tbl' iam_role default delimiter '|' region 'us-east-1';

This is a problem because Athena tables can only be defined on prefixes and not individual objects. Fortunately, that's only the case for the 10GB dataset. The other sizes reference prefixes, which contain multiple files, and that makes setting up tables for these easier.

To resolve this issue for the 10GB dataset, we have to copy the data to an S3 bucket that we control and store the individual files in separate prefixes. That makes the process for the 10GB workflow more complicated than the larger datasets, but didn't find a better way to work around this. As a result we'll have two DAGs later, one, that handles the smaller dataset, and another, that can handle any of the other sizes.

Ah, DAG - this is the first time I'm mentioning it after the headline - I've written the provisioning process as a DAG for Airflow, mostly because I felt like it. Airflow is not usually meant to be (mis)used for these one-off processes. If you're new to Airflow, maybe check out this introduction to Airflow on AWS that we published a while ago. From now on, I'm going to assume some familiarity with it.

The DAGs I'll be talking about are available on Github, and I won't be covering every implementation detail as they're a bit verbose. We'll first focus on the simple case, which means the bigger datasets, because the more complex case is just an extension of these.

For the 100GB+ datasets, I've written a DAG that accepts a few parameters and then creates Athena tables based on the raw data in the AWS S3 Bucket, executes a CTAS query to convert this data into parquet, stored in our bucket, and finally deletes the intermediate table for the raw data (which can be disabled). In the GUI, the DAG looks something like this:

As you can see, I've employed task groups to visually group the individual processing steps. Since this is a DAG I expect to be triggered manually, I've also made it configurable through Params.

with DAG(
    dag_id="create_larger_tpch_dataset",
    dag_display_name="Create a large TPC-H dataset",
    default_args={"retries": 1, "retry_delay": timedelta(minutes=5)},
    params={
        "size": Param(
            default="100GB",
            enum=["100GB", "3TB", "30TB"],
            description="Size of the dataset to create, this is based on the unoptimized version of the data.",
            title="Dataset Size (unoptimized)",
        ),
# ...
        ),
        "s3_prefix": Param(
            default="tpch_100gb/",
            description="Prefix to store the data under. Must be either / (root of bucket) or a string ending in slash, e.g. some/prefix/",
            type="string",
            title="S3 Prefix",
            pattern=r"(^\/$|^[^\/].*\/$)",
        ),
# ...
        "athena_output_location_s3_uri": Param(
            default="s3://aws-athena-query-results-account-eu-central-1/",
            type="string",
            pattern=r"^s3:\/\/[a-z0-9-_]*\/.*$",
            title="Athena Result location",
            description="S3 URI to store the Athena results under, e.g. s3://my-bucket/and_prefix/.",
        ),
    },
) as dag:

Airflow uses the parameters to provide this convenient GUI to start the DAG.

As you can see, the DAG allows you to select a database in Athena where you want to create your tables, the storage location for the optimized data, and a temporary Athena result location. Selecting a workgroup is also mandatory to make this work. You can also disable dropping the tables for the raw data here, but you should be aware that they reference data in us-east-1, and querying them may incur more charges than you expect.

If you only need a subset of the tables, you'll have to edit the DAG definition yourself. You'll find a TABLE_NAMES list near the top of the file - just comment out whichever tables you don't need.

Depending on which dataset size you picked, this may run for a few minutes, but afterward, you should see eight new tables in the database you entered. Later, we'll run benchmark queries to ensure everything works as expected. But first, let's have a look at the DAG for the 10GB of data. As mentioned above, we need to copy the data into one of our S3 buckets first so that Athena can use it as a table.

Here I ran into a problem where the S3CopyObjectOperator can natively only copy objects up to 5GB in size, so I had to write an extension of that operator, which I documented in this blog post.

Aside from copying the data to our bucket, we also needed an additional cleanup step in the end to remove it if the user chooses. The raw data is uncompressed, so we can save a few dollars here.

The parameters for this DAG are very similar. I didn't need the size selector, and aside from that, only a few descriptions differ, so I'm not going to show the GUI again. Instead, let's test that our data is accessible by running a few benchmark queries from the aforementioned AWS repository. Unfortunately, there are some syntax differences between Athena SQL and Redshift SQL, so not all of them can be used as-is, but here's an example that works on the 100GB dataset (which is about 22GB as parquet):

So, how can you use these? I've put the DAGs on Github, you can just download them and add them to your DAG directory, where Airflow should pick them up. Then you trigger the DAG you want through the GUI after configuring the paramters and you're good to go.

There are some more things I should mention before we're done. The data appears to be created using the v2.18 generator, and a new major version is out, but that shouldn't be too much of an issue since this is intended to be used for demos or practicing anyway. Additionally, we're relying on AWS keeping the data around in their S3 bucket, but they've been doing that for years, so I'm fairly confident this is stable.

In conclusion, I've shown you how to provision the TPC-H dataset in Athena using Airflow. Now go and build on top of it!

— Maurice

Title Photo by Mika Baumeister on Unsplash