DEV Community: André

Leveraging DynamoDB Split for Heat: an Architecture Suggestion

André — Mon, 01 Jun 2020 18:48:12 +0000

Disclaimer: I'm not affiliated with AWS and this is not technical advice. In case you consider using this pattern in production, please be aware the core feature is very briefly documented and there is actually no performance guarantee.

According to the docs, DynamoDB's auto-split, split-for-heat or Adaptive Capacity auto-split is a (briefly documented) feature that "rebalances your partitions such that frequently accessed items don't reside on the same partition".

After performing some tests to have a better understanding on how this feature actually works, and to acknowledge that it isn't a replacement for write sharding, I came up with this architecture as a way to improve read costs for, maybe, some use cases.

Lazy Sharding

Basically, the idea here is to only write-shard the partition if our requests are being throttled. If they are, run a de-sharding strategy to rewrite the items to a single partition, while DynamoDB takes care of the repartitioning. When reading, also perform an attemptive read on the main partition first, only performing a sharded read as a second attempt.

Alternatively, for cases where the reads are disproportionally high, not allowing the partitioning to happen based on write velocities, similar de-sharding jobs can be run, so attemptive reads on a single partition can be done in order to save RCUs.

Is it Worth it? Would it work?

Honestly, in addition to the fact that there's no performance guarantee on the underlying feature, I'm not sure if there are many use cases for this. So, please tell me what you think, leave your feedback!

--
André

Why You Can't (yet) Deprecate Write Sharding on DynamoDB

André — Mon, 01 Jun 2020 18:45:39 +0000

Disclaimer: I'm not affiliated with AWS and this is not technical advice.

Motivation

A couple of months ago, revisiting DynamoDB's documentation, this piece caught my attention:

Immediately after that, I remembered all the thousand times I watched Rick Houlihan's talks, specially the parts where he explains how to make use of write sharding on your DynamoDB table designs, in order to maximize throughput by avoiding hot partitions.

It was clear to me that this piece of documentation was about a feature that presumably addressed the same issue as write sharding (hot partitions), but in an adaptive way. Although the documentation doesn't tell us much, it does make a bold promise: to automatically repartition your items across as many nodes as needed, down to a single-item partition. It looked like a very handy feature, since hot partitions are a real thing for highly scalable applications, and it's something you really need to consider before committing your table design. Also, not having to deal with write sharding means not pushing it down the pipe, forcing the application layer to handle the bundled complexity.

Most importantly, write sharding strategies increase your throughput capacity consumption. Since this "auto-split" feature comes at no cost, making use of it means saving money.

Naturally, an adaptive solution at the service layer screams for a loosely coupled, fault tolerant system (which is not necessarily the case for write sharding, where the splitting happens on your side, under your control). But then again, what's the matter? The problem we're trying to address concerns applications running at high scale, and these are most likely built on top of those best practices, anyway.

Too good to be true?

This whole thing got me intrigued. How come this feature exists and no one talks about it? So I started asking questions:

// Detect dark theme var iframe = document.getElementById('tweet-1250389357967654912-462'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1250389357967654912&theme=dark" }

// Detect dark theme var iframe = document.getElementById('tweet-1252703607931240449-952'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1252703607931240449&theme=dark" }

// Detect dark theme var iframe = document.getElementById('tweet-1258441534053257216-45'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1258441534053257216&theme=dark" }

Kirk Kirkconnell's answers aside, which confirmed that the feature existed and there was an upcoming documentation overhaul that would make things clearer, no one else had an actual answer. In a recent tweet, I even tried to tease Rick Houlihan himself, but had no luck there.

edit: A while later, Rick attentively answered all my questions. Thanks, Rick!

At this point, I was already getting paranoid.

I needed to do something about it.

Tests

I finally dedicated some time to build a CDK app to test this. If you're interested, check out this blog post for a walkthrough.

Considerations

A few things to consider before load testing a DynamoDB table:

There's a default limit of 40k WCUs and 40k RCUs.
Tables running in On-Demand capacity mode have an initial "previous peak" value of 2k WCUs or 6k RCUs.
The above means that the initial throughput limit for an On-Demand table is 4k WCUs or 12k RCUs (or a linear combination of the two, eg.: 0 WCUs and 12k RCUs).
The new "previous peak" is established approximately 30 minutes after the new peak is reached, then effectively doubling the throughput limit.

It's also relevant to remember what exactly we are testing. We want to prove the hypothesis of performing read and write operations in a single table, using a single DynamoDB Partition Key, at a sustained throughput that has to be higher than the advertised limit (1k WCUs / 3k RCUs per partition).

Smoke Testing

The first thing I wanted to test, was if this feature existed at all. And I thought that if it was to exist, it would have to at least support reads.

In order to make read requests, I needed to populate the table. So, I triggered the execution of an insert batch, making sure the load didn't reach the throughput limit of the partition.

Workers	Duration	Load	Interval	Target Throughput	Total Capacity
3 units	600 secs	300 items	1 sec	900 WCU	540k WCU

Insert Batch Completed in Step Functions

DynamoDB Write Capacity Widget

With the items in place, I ran another batch, this time with a target throughput of about 6k RCUs, which is twice the partition limit:

Workers	Duration	Load	Interval	Target Throughput	Total Capacity
20 units	600 secs	300 items	0.8 sec	6000 RCU	3600k RCU

DynamoDB Read Capacity Widget

As you can see, we could reach the target throughput. This is our custom CloudWatch Dashboard Widget after performing both steps of this test:

CloudWatch Dashboard Widget

However, we have experienced some throttling. DynamoDB Metrics says we had around 1.40% of throttled reads.

DynamoDB Throttled Read Events Widget

Since the node limit is 3k RCU, populating the table at around 900 WCU might have split our data into two nodes, allowing us to reach 6k RCUs.

Knowing how DynamoDB throughput limits work at the table level, I thought that maybe we have reached a new plateau at 6k RCUs. This would explain this marginal throttling rate.

Smoking Test: Higher Load

Then, I ran another test. Starting again with a new table, populated within the partition limits. This time though, I'll try to reach something just below 10k RCUs. My goal here is to find another plateau and establish a pattern on the feature behaviour.

Workers	Duration	Load	Interval	Target Throughput	Total Capacity
33 units	600 secs	300 items	0.9 sec	9900 RCU	5940k RCU

Throttled at 6k RCU, then reached the new peak.

As suspected, we indeed had reached a plateau at 6k RCU. It took around 8 minutes for the auto-split to kick in and repartition our data, so we could again reach another peak. Then we could run for the remaining 2 minutes without throttling.

Surpassing the 6k plateau, peaking at around 9k RCU throttle-free.

Now let's see what happens when trying to read at 12k RCUs on the same table. The idea is to test for a new plateau and see if the feature can handle a sustained throughput at around the previous peak without throttling.

Workers	Duration	Load	Interval	Target Throughput	Total Capacity
42 units	600 secs	300 items	0.9 sec	12600 RCU	7200k RCU

Just like we have guessed, whatever the "auto-split" feature did, it's there. A relatively small throttle count happened when the throughput surpassed the 12k plateau, either because of the On-Demand initial table limit or because we have reached another plateau:

No throttling at 6k RCU

Testing the Plateau Pattern in a Real World Scenario

Now that we have a better hypothesis on how this feature works, let's try a real world scenario. Let's imagine that an application needs to support the case where a new Partition Key suddenly starts to suffer a big amount of reads and writes. Since it's a new partition, there's no chance it was already re-partitioned.

Our write workload will target 2.5k WCUs, and our read workload will target 6k RCUs

Workers	Duration	Load	Interval	Target Throughput	Total Capacity
5 units	600 secs	500 items	1 sec	2500 WCU	1500k WCU
20 units	600 secs	300 items	0.9 sec	6000 RCU	3600k RCU

Throttling fest!

As you can see, many bad things happened here. As we could have guessed, the re-partitioning isn't instantaneous, and it's unclear what exactly makes it happen.

Throttled Events vs Consumed WCUs

Throttled Events vs Consumed RCUs

It's also interesting to see that the throughput capacity randomly reaches another plateau as if the feature didn't follow a pattern over time.

Another Test Case

Now let's imagine an application that needs its partitions to have their throughput capacity linearly increasing over time. Based on the tests performed so far, I'll run a test that simultaneously reads and writes to a new partition at an initial velocity of 100WCUs / 300RCUs and accelerates 15% every minute.

Workers	Duration	Load	Interval	Target Throughput	Total Capacity
5 units	1200 secs	20-277 items	1 sec	100-1385 WCU	600k WCU
20 units	1200 secs	15 items	1 sec	300-4230 RCU	1830k RCU

Much better, but still some throttling.

If we look at all the tests so far, besides the fact that the split seems to be happening randomly across the test duration, we can see that the capacity is always increased by 1k WCUs each time a new plateau is reached. This behaviour makes me think that regardless of how much throttling is happening, the feature acts by adding a single node to the partition.

Running another batch on the same table and partition, now loading 6000 RCU + 2000 WCU steadily (which is actually 4 times the parition capacity, since the operations are running simultaneously), we can see that there's no throttling.

Workers	Duration	Load	Interval	Target Throughput	Total Capacity
5 units	600 secs	400 items	1 sec	2000 WCU	1200k WCU
20 units	600 secs	300 items	1 sec	6000 RCU	3600k RCU

No throttling!

It looks like the repartitioning is somewhat persistent.

Settling down

In the next test, we'll apply the exact same load as the previous batch, which is again 4 times the initial partition capacity. But this time, we'll do it for 30 minutes in a new partition. The idea is to validate the behavior we could infer from last test. We'll do it on the same table, so the table limits don't masquerade the results.

Workers	Duration	Load	Interval	Target Throughput	Total Capacity
5 units	1200 secs	400 items	1 sec	2000 WCU	1200k WCU
20 units	1200 secs	300 items	1 sec	6000 RCU	3600k RCU

Random plateau changes

Again, as you can see in the graph above, the time it takes for the partition to be split is pretty random, and it certainly depends on many variables we cannot control.

To make things even more clear, we'll do another batch on the same table and new partition, but now with a load of 5k WCUs for over an hour.

Workers	Duration	Load	Interval	Target Throughput	Total Capacity
10 units	4800 secs	400 items	1 sec	4000 WCU	19200k WCU

Random intervals, 1k WCUs per change.

Test Conclusions

It looks like DynamoDB, in fact, has a working auto-split feature for hot partitions. Also, there are reasons to believe that the split works in response to a high usage of throughput capacity on a single partition, and that it always happens by adding a single node, so that the capacity is increased by 1kWCUs / 3k RCUs each time. The "split" also appears to be persistent over time.

Final Thoughts

The "auto-split" feature seems to be a best effort to accommodate unpredicted loads that require more throughput capacity than available in the partition.

It's important to note that the feature is very briefly described on the official docs, and there's no promise on how it actually performs. If I had to guess, I would say this is something still being worked on. My instincts say it'll be improved and launched as an "auto-sharding" feature when it's mature enough.

In the meantime though, since this clearly isn't a replacement for write sharding and considering the auto-split as proven to be persistent over time, is there any other way we could make use of it?

For some applications that have to make use of write sharding to increase the throughput capacity of their partitions, here is an architecture suggestion:

Lazy Sharding

For more details and hopefully a discussion about this suggestion, check this post.

Thanks for reading!

--
André

How to Load Test Your DynamoDB Table Using a CDK Serverless App

André — Mon, 01 Jun 2020 18:42:48 +0000

I recently needed to clarify an understanding about a DynamoDB feature and, in the lack of other means, decided to run some tests to see how it would perform. In order to do so, I built a CDK app featuring:

Fargate Workers
CloudWatch Dashboard
Step Functions State Machine
Execution Manager

Here is a walkthrough:

Disclaimer: Unless you're testing this with very light loads, deploying this App will cost you money. I'm not responsible for that.

Bootstrap a new CDK app

Install CDK: npm install -g aws-cdk
Create a new CDK app:

mkdir dynamodb-load-test && cd dynamodb-load-test
cdk init --language typescript

Construct Interfaces

lib/dynamodb-load-test-stack.ts

interface DynamoDBLoadTestProps extends cdk.StackProps {
  /**
   *  Configuration for the INSERT worker.
   *
   * @default
   */
  readonly insertWorkerProps: WorkerProps;

  /**
   *  Configuration for the READ worker.
   *
   * @default
   */
  readonly readWorkerProps: WorkerProps;

  /**
   *  Configuration for the UPDATE worker.
   *
   * @default
   */
  readonly updateWorkerProps: WorkerProps;

  /**
   * Execution input
   */
  readonly executionInput?: executionInputProps;
}

interface executionInputProps {
  /**
   * The state to resume to.
   *
   */
  readonly resumeTo: string;

  /**
   * If should run next batch.
   * "YES" or "NO"
   *
   */
  readonly runNext: string;

  /**
   * Other inputs
   *
   */
  readonly [name: string]: string | undefined;
}

interface WorkerProps {
  /**
   * Total number of copies for this worker.
   *
   * @default 5
   */
  readonly copies: number;

  /**
   * Total number of iterations that will happen after each interval.
   *
   * @default 600
   */
  readonly iterations?: number;

  /**
   * Interval on top of processing time in milliseconds.
   *
   * @default 1000
   */
  readonly interval?: number;

  /**
   * Number of capacity units (WCU or RCU) per iteration.
   *
   * @default 300
   */
  readonly load?: number;

  /**
   * Rate on which to increment the load (e.g: 0.3 for 30%).
   *
   * @default 0.0
   */
  readonly  increment?: number;

  /**
   * Limit to use when specifying an increment rate. Use -1 for no limit.
   *
   * @default -1
   */
  readonly loadLimit?: number;

  /**
   * Number of intervals to trigger an increment (e.g: if the interval is 1000ms, set it
   * to 60 to have an increment per minute)
   *
   * @default 60
   */
  readonly incrementTime?: number;
}

Adding our victim (the Table)

This is pretty straight-forward using the L1 construct available at the official CDK construct library.

npm install @aws-cdk/aws-dynamodb

import * as ddb from '@aws-cdk/aws-dynamodb';

    // Table
    const table = new ddb.Table(this, "WillItThrottle", {
      partitionKey: {name: "PK", type: ddb.AttributeType.STRING},
      sortKey: {name: "SK", type: ddb.AttributeType.STRING},
      billingMode: ddb.BillingMode.PAY_PER_REQUEST,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });

Workers

For the workers, I opted to use Fargate, the AWS serverless offer for Docker container workloads. Fargate is a perfect fit for ephemeral tasks / test workloads as we don't need to worry about provisioning servers.

npm install @aws-cdk/aws-ec2
npm install @aws-cdk/aws-ecs
npm install @aws-cdk/aws-iam

import * as ec2 from '@aws-cdk/aws-ec2';
import * as ecs from '@aws-cdk/aws-ecs';
import * as iam from '@aws-cdk/aws-iam';

    // Fargate Cluster and VPC
    const vpc = new ec2.Vpc(this, 'Vpc', {maxAzs: 1});
    const cluster = new ecs.Cluster(this, 'FargateCluster', {vpc});

    // ECS Task Role with access to Table
    const taskRole = new iam.Role(this, "TestWorkerRole", {
      assumedBy: new iam.ServicePrincipal("ecs-tasks.amazonaws.com"),
    });
    table.grantReadWriteData(taskRole);

    // Worker
    const testWorker = new TestWorker(this, "FargateWorker", {
      tableName: table.tableName,
      taskRole,
    });

All workers will use the same image that contains three different ts-node scripts, one for each task type (insert, read, update). As you might have noticed, the workers' code was isolated in a separate construct.

  interface TestWorkerProps {
  readonly taskRole: iam.IRole,
  readonly tableName: string,
}

class TestWorker extends cdk.Construct {
  private readonly _taskDefinition: ecs.FargateTaskDefinition;
  private readonly _container: ecs.ContainerDefinition;

  constructor(scope: cdk.Construct, id: string, props: TestWorkerProps) {
    super(scope, id);

    this._taskDefinition = new ecs.FargateTaskDefinition(this, "DDBTestWorkerTask", {
      taskRole: props.taskRole,
      memoryLimitMiB: 1024,
    })

    this._container = new ecs.ContainerDefinition(this._taskDefinition, "DDBTestWorkerTaskContainer", {
      taskDefinition: this._taskDefinition,
      logging: ecs.LogDriver.awsLogs({
        streamPrefix: "DDBTestWorkerTaskContainer"
      }),
      image: ecs.ContainerImage.fromAsset("./lib/images/worker"),
      environment: {
        TABLE_NAME: props.tableName,
      }
    });
  }

  public get taskDefinition(): ecs.FargateTaskDefinition {
    return this._taskDefinition;
  }

  public get container(): ecs.ContainerDefinition {
    return this._container;
  }
}

Notice how easy it is to build custom Docker images in the context of your CDK App by using the fromAsset() method. It builds your image and updates to it a repository, linking the reference to your deployment behind the scene.

Since all the workers have the same Task Definition, we'll need to make them behave differently by passing parameters at execution time in the form of "container overrides".

Orchestration and Iteration

In order to orchestrate the test batches, a Step Functions State Machine was used. It was built chaining Task States that will define the ultimate behaviour our workers, such as the velocity and duration of the load, as well as the type of the workload (insert, read or update).

npm install @aws-cdk/aws-stepfunctions
npm install @aws-cdk/aws-stepfunctions-tasks

import * as sfn from '@aws-cdk/aws-stepfunctions';
import * as sft from '@aws-cdk/aws-stepfunctions-tasks';

// Insert Task
    const insertWorkerTask = new sft.RunEcsFargateTask({
      taskDefinition: testWorker.taskDefinition,
      integrationPattern: sfn.ServiceIntegrationPattern.SYNC,
      cluster,
      containerOverrides: [
        {
          containerName: testWorker.container.containerName,
          environment: [
            {
              name: "DURATION",
              value: (props?.insertWorkerProps.iterations?.toFixed(0)) || "600", // INSERT duration 
            },
            {
              name: "LOAD",
              value: (props?.insertWorkerProps.load?.toFixed(0)) || "300", // INSERT load (items / interval)
            },
            {
              name: "INTERVAL",
              value: (props?.insertWorkerProps.interval?.toFixed(0)) || "1000", // INSERT interval 
            },
            {
              name: "INCREMENT",
              value: (props?.insertWorkerProps.increment?.toFixed(2)) || "0", // INSERT incremental rate (e.g: 0.3 for 30% each INC_TIME)
            },
            {
              name: "LOAD_LIMIT",
              value: (props?.insertWorkerProps.loadLimit?.toFixed(0)) || "-1", // INSERT maximum load limit (-1 for no limit)
            },
            {
              name: "INC_TIME",
              value: (props?.insertWorkerProps.incrementTime?.toFixed(0)) || "60", // INSERT time to apply INCREMENT 
            },
          ],
          command: ["ts-node", '/opt/insert.ts'] // type of load
        }
      ]
    });

    // Read Task
    const readWorkerTask = new sft.RunEcsFargateTask({
      taskDefinition: testWorker.taskDefinition,
      integrationPattern: sfn.ServiceIntegrationPattern.SYNC,
      cluster,
      containerOverrides: [
        {
          containerName: testWorker.container.containerName,
          environment: [
            {
              name: "DURATION",
              value: (props?.readWorkerProps.iterations?.toFixed(0)) || "600", // READ duration
            },
            {
              name: "LOAD",
              value: (props?.readWorkerProps.load?.toFixed(0)) || "300", // READ load (items / interval)
            },
            {
              name: "INTERVAL",
              value: (props?.readWorkerProps.interval?.toFixed(0)) || "1000", // READ interval 
            },
            {
              name: "INCREMENT",
              value: (props?.readWorkerProps.increment?.toFixed(2)) || "0", // READ incremental rate (e.g: 0.3 for 30% each INC_TIME)
            },
            {
              name: "LOAD_LIMIT",
              value: (props?.readWorkerProps.loadLimit?.toFixed(0)) || "-1", // READ maximum load limit (-1 for no limit)
            },
            {
              name: "INC_TIME",
              value: (props?.readWorkerProps.incrementTime?.toFixed(0)) || "60", // READ time to apply INCREMENT 
            },
          ],
          command: ["ts-node", '/opt/read.ts'] //insert, update, read
        }
      ]
    });

    // Update Task
    const updateWorkerTask = new sft.RunEcsFargateTask({
      taskDefinition: testWorker.taskDefinition,
      integrationPattern: sfn.ServiceIntegrationPattern.SYNC,
      cluster,
      containerOverrides: [
        {
          containerName: testWorker.container.containerName,
          environment: [
            {
              name: "DURATION",
              value: (props?.updateWorkerProps.iterations?.toFixed(0)) || "600", // UPDATE duration
            },
            {
              name: "LOAD",
              value: (props?.updateWorkerProps.load?.toFixed(0)) || "300", // UPDATE load (items / interval)
            },
            {
              name: "INTERVAL",
              value: (props?.updateWorkerProps.interval?.toFixed(0)) || "1000", // UPDATE interval 
            },
            {
              name: "INCREMENT",
              value: (props?.updateWorkerProps.increment?.toFixed(2)) || "0", // UPDATE incremental rate (e.g: 0.3 for 30% each INC_TIME)
            },
            {
              name: "LOAD_LIMIT",
              value: (props?.updateWorkerProps.loadLimit?.toFixed(0)) || "-1", // UPDATE maximum load limit (-1 for no limit)
            },
            {
              name: "INC_TIME",
              value: (props?.updateWorkerProps.incrementTime?.toFixed(0)) || "60", // UPDATE time to apply INCREMENT 
            },
          ],
          command: ["ts-node", '/opt/update.ts'] //insert, update, read
        }
      ]
    });

To define the horizontal scale of the batches, nothing better than Step Functions Parallel state.

    // Inserts in Parallel
    const insertData = new sfn.Parallel(this, "InsertData", {
      resultPath: "DISCARD",
    });
    for (let x = 0; x < (props?.insertWorkerProps.copies || 5); x++) {
      insertData.branch(new sfn.Wait(this, "INSERT " + x.toString(), {time: sfn.WaitTime.duration(cdk.Duration.seconds(x))})
        .next(
          new sfn.Task(this, "InsertWorker " + x.toString(), {
            task: insertWorkerTask,
          })));
    }
    // Reads in Parallel
    const readData = new sfn.Parallel(this, "ReadData", {
      resultPath: "DISCARD",
    });
    for (let x = 0; x < (props?.readWorkerProps.copies || 5); x++) {
      readData.branch(new sfn.Wait(this, "READ " + x.toString(), {time: sfn.WaitTime.duration(cdk.Duration.seconds(x))})
        .next(
          new sfn.Task(this, "ReadWorker " + x.toString(), {
            task: readWorkerTask,
          })));
    }
    // Updates in Parallel
    const updateData = new sfn.Parallel(this, "UpdateData", {
      resultPath: "DISCARD",
    });
    for (let x = 0; x < (props?.updateWorkerProps.copies || 5); x++) {
      updateData.branch(new sfn.Wait(this, "UPDATE " + x.toString(), {time: sfn.WaitTime.duration(cdk.Duration.seconds(x))})
        .next(
          new sfn.Task(this, "UpdateWorker " + x.toString(), {
            task: updateWorkerTask,
          })));
    }

To manage the execution iterations, this CDK construct was used. It takes a State Machine and an input as props and gives us a nice iteration routine, conveniently displaying a link to the current execution details on Step Functions console. I also added a Pass state (endState), so it was possible to perform "dry runs" and two Choice states, so I could stop the execution after a certain state.

npm install cdk-execution-manager

    // State Machine
    const definition = insertData
      .next(new sfn.Choice(this, "QuitAfterInsert?").when(sfn.Condition.stringEquals("$.runNext", "NO"), endState)
        .otherwise(readData.next(new sfn.Choice(this, "QuitAfterRead?").when(sfn.Condition.stringEquals("$.runNext", "NO"), endState)
          .otherwise(updateData))));

    const deployment = new DeploymentManager(this, "TestDeployment", {
      stateMachineDefinition: definition,
      executionInput: props?.executionInput || {
        resumeTo: insertData.id,
        runNext: "YES"
      }
    });

Monitoring

CloudWatch dashboard widget in action

Besides the default DynamoDB metrics, it's nice to have other metrics / dashboards more specific to what we're testing. This can be done by configuring a CloudWatch Dashboard.

npm install @aws-cdk/aws-cloudwatch

    // CW Dashboard
    const dash = new DynamoDBTestCWDashboard(this, "WillItThrottleDashBoard", {
      tableName: table.tableName,
    });

...

class DynamoDBTestCWDashboard extends cdk.Construct {
  constructor(scope: cdk.Construct, id: string, props: { tableName: string }) {
    super(scope, id);

    const dash = new cw.Dashboard(this, id, {
      dashboardName: "DynamoDB-AC-Auto-Split-Tests",
    });

    const readThrottleEvents = new cw.Metric({
      namespace: 'AWS/DynamoDB',
      metricName: 'ReadThrottleEvents',
      period: cdk.Duration.minutes(1),
      statistic: "sum",
      dimensions: {
        TableName: props.tableName,
      }
    });

    const writeThrottleEvents = new cw.Metric({
      namespace: 'AWS/DynamoDB',
      metricName: 'WriteThrottleEvents',
      period: cdk.Duration.minutes(1),
      statistic: "sum",
      dimensions: {
        TableName: props.tableName,
      }
    });

    const consumedReadCapacityUnits = new cw.Metric({
      namespace: 'AWS/DynamoDB',
      metricName: 'ConsumedReadCapacityUnits',
      period: cdk.Duration.minutes(1),
      statistic: "sum",
      dimensions: {
        TableName: props.tableName,
      }
    });

    const consumedWriteCapacityUnits = new cw.Metric({
      namespace: 'AWS/DynamoDB',
      metricName: 'ConsumedWriteCapacityUnits',
      period: cdk.Duration.minutes(1),
      statistic: "sum",
      dimensions: {
        TableName: props.tableName,
      }
    });

    const consumedRCU = new cw.MathExpression({
      expression: "r/60",
      usingMetrics: {
        r: consumedReadCapacityUnits,
      },
      label: "Consumed RCUs",
      color: "#114477",
      period: cdk.Duration.minutes(1),
    });

    const consumedWCU = new cw.MathExpression({
      expression: "w/60",
      usingMetrics: {
        w: consumedWriteCapacityUnits,
      },
      label: "Consumed WCUs",
      color: "#117744",
      period: cdk.Duration.minutes(1),
    });

    const graph = new cw.GraphWidget({
      title: "DynamoDB Test",
      right: [consumedRCU, consumedWCU],
      left: [readThrottleEvents, writeThrottleEvents],
      rightAnnotations: [
        {label: "RCU Node Limit", color: "#114477", value: 3000},
        {label: "WCU Node Limit", color: "#117744", value: 1000},
      ]
    })

    dash.addWidgets(graph);
  }

}

Here is the App's (probably updated) source code, including the test workers.

Thanks for reading!

--
André