Grokking DynamoDB with TypeScript

DynamoDB is a fully managed NoSQL database service provided by AWS. It is highly available, scalable, and a great fit for applications that need to handle large amounts of data with low latency. DynamoDB is a key-value document database, meaning that it is schema-less and can store any kind of data. It is also a serverless service, so you don’t have to worry about managing and scaling servers.

In this take, we will model data in DynamoDB using TypeScript. We will start by modeling a simple table with a partition key and some attributes. Then, we will model another table with a composite primary key using a partition key and a sort key. We will also explore secondary indexes and conditional expressions.

Setup and Prerequisites

Our data model will consist of a made-up open-world RPG game. We will model a table for characters and a table for quests. The characters table will have a simple partition key, and the quests table will have a composite primary key.

The access patterns will dictate how we will model this data based on use cases for the game.

As a prerequisite, you should have:

• An AWS account with the AWS CLI installed and configured
• Node.js installed
• npm installed

You can also clone the GitHub repository for this project.

To follow along with the code examples, create a TypeScript project using the following commands:

mkdir node-dynamodb-playground
cd node-dynamodb-playground
npm init -y
npm i typescript @aws-sdk/client-dynamodb @aws-sdk/util-dynamodb ulid
npx tsc --init

Be sure to add the following to your package.json file:

{
  "scripts": {
    "build": "tsc"
  }
}

Ready? Let’s get started!

What Are Access Patterns in DynamoDB?

In DynamoDB, access patterns are the different ways to access your data. They are the queries and scans that you will perform on your tables. These must come directly from your application’s use cases.

For our RPG game, we will have the following access patterns:

• Get character by username
• Fetch inventory for a character
• Fetch characters by guild
• Get all quests for a character
• Fetch completed quests for a character

Note: The key to modeling data in DynamoDB is to understand your access patterns first and then model your data accordingly.

A common misconception about NoSQL databases is that they are schema-less and, therefore, flexible. While it is true that you can store any kind of data in DynamoDB, it is important to know your access patterns. If you want to keep latencies low, you need to optimize your data model for your access patterns, which means the schema is actually not very flexible.

Now that we have our access patterns, let’s start modeling our data.

Partition Keys

Using the AWS CLI tool, create a new table named characters with a simple partition key.

aws dynamodb create-table \
  --table-name characters \
  --attribute-definitions AttributeName=username,AttributeType=S \
  --key-schema AttributeName=username,KeyType=HASH \
  --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1

Then, let’s create a TypeScript interface to model our character.

// models/character.ts

export interface Character {
  username: string;
  class: string;
  guild: string;
  inventory: Record<string, number>;
  total_play_time: number;
}

Now, add a character to the table via TypeScript.

// add-character.ts

import { DynamoDBClient, PutItemCommand } from "@aws-sdk/client-dynamodb";
import { marshall } from "@aws-sdk/util-dynamodb";

import type { Character } from "./models/character";

const client = new DynamoDBClient();

const addCharacter = async (character: Character): Promise<void> => {
  await client.send(
    new PutItemCommand({
      TableName: "characters",
      Item: marshall(character),
    })
  );
};

void addCharacter({
  username: "beautifulcoder",
  class: "Mage",
  guild: "Hacker",
  inventory: {
    "Mechanical Keyboard": 1,
    "Kaihl Switches": 100,
    Coffee: 100000,
  },
  total_play_time: 3600,
});

Note that we are using the marshall function from @aws-sdk/util-dynamodb to convert our JavaScript object to a DynamoDB record. In this way, we can use TypeScript interfaces to model our data and then convert this to a DynamoDB record.

DynamoDB supports the following data types:

• Scalar types: Number, String, Binary, Boolean, and Null
• Document types: Array and JavaScript Object
• Set types: Number Set, String Set, and Binary Set

Note: Set types must be unique and do not support duplicates.

The TypeScript interface strictly adheres to our access patterns and DynamoDB data types. We can get a character by username, pulling up their inventory and guild within a single round-trip. One recommendation is to first define the access patterns and then write the TypeScript interfaces to model the data.

Optimizing access patterns will also reduce the number of round-trips to the database, which will help keep latencies low.

Composite Keys

Now, let’s model another table with a composite primary key, using a partition key and a sort key. We will model a table for quests with a composite primary key. This time, we will use a ULID as the sort key to help model and access our data.

aws dynamodb create-table \
  --table-name quests \
  --attribute-definitions AttributeName=username,AttributeType=S AttributeName=quest_id,AttributeType=S \
  --key-schema AttributeName=username,KeyType=HASH AttributeName=quest_id,KeyType=RANGE \
  --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1

To model a quest, we will use the following TypeScript interface:

// models/quest.ts

export interface Quest {
  username: string;
  quest_id: string;
  quest_name: string;
  quest_started_at: string;
  quest_completed_at?: string;
  checkpoints: number[];
  gold: number;
}

The quest_completed_at attribute remains optional, because the quest may never be completed. This is one of the benefits of using a NoSQL database. You can store any kind of data and reduce any unnecessary attributes.

Now, add a quest to the table via TypeScript.

// add-quest.ts

import { DynamoDBClient, PutItemCommand } from "@aws-sdk/client-dynamodb";
import { marshall } from "@aws-sdk/util-dynamodb";
import { ulid } from "ulid";

import type { Quest } from "./models/quest";

const client = new DynamoDBClient();

const addQuest = async (quest: Quest): Promise<void> => {
  await client.send(
    new PutItemCommand({
      TableName: "quests",
      Item: marshall(quest),
    })
  );
};

const now = new Date();
const twoWeeksFromNow = new Date(Date.now() + 12096e5);
const minuteFromNow = new Date(Date.now() + 60000);

void addQuest({
  username: "beautifulcoder",
  quest_id: ulid(now.getTime()),
  quest_name: "Sole Survivor",
  quest_started_at: now.toISOString(),
  quest_completed_at: twoWeeksFromNow.toISOString(),
  checkpoints: [1, 2, 3, 4, 5],
  gold: 1000,
});

void addQuest({
  username: "beautifulcoder",
  quest_id: ulid(minuteFromNow.getTime()),
  quest_name: "A Lost Cause",
  quest_started_at: now.toISOString(),
  checkpoints: [6, 7, 8, 9, 10],
  gold: 10000,
});

Because we use a ULID as the sort key, DynamoDB will sort the quests by the time they are created. This will help us fetch all quests for a character and easily filter the data based on the time the quest was created.

Secondary Indexes

So far, we have modeled our data based on most of our access patterns. However, we have not yet modeled the access pattern to fetch our characters by guild. We can use a global secondary index to help query the data without a table scan.

aws dynamodb update-table \
  --table-name characters \
  --attribute-definitions AttributeName=guild,AttributeType=S AttributeName=username,AttributeType=S \
  --global-secondary-index-updates file://gsi-guild-username.json

Create a file named gsi-guild-username.json with the following JSON content:

[
  {
    "Create": {
      "IndexName": "gsi-guild-username",
      "KeySchema": [
        {
          "AttributeName": "guild",
          "KeyType": "HASH"
        },
        {
          "AttributeName": "username",
          "KeyType": "RANGE"
        }
      ],
      "Projection": {
        "ProjectionType": "ALL"
      },
      "ProvisionedThroughput": {
        "ReadCapacityUnits": 1,
        "WriteCapacityUnits": 1
      }
    }
  }
]

This global secondary index works much like a partition key and sort key, but with a different primary key. DynamoDB will create a new index with the same data and replicate the data from the main table. All writes must still flow through the primary table because secondary indexes are read-only.

Lastly, to fetch the quests that a character completes, we can filter our data by using a global secondary index. This will maintain the same primary key as the main table, but with a different sort key. This will also exclude any uncompleted quests from the secondary index. DynamoDB will optimize the reads because it does not send any irrelevant data in the queries.

aws dynamodb update-table \
  --table-name quests \
  --attribute-definitions AttributeName=username,AttributeType=S AttributeName=quest_completed_at,AttributeType=S \
  --global-secondary-index-updates file://gsi-username-quest_completed_at.json

Create a file named gsi-username-quest_completed_at.json with the following content:

[
  {
    "Create": {
      "IndexName": "gsi-username-quest_completed_at",
      "KeySchema": [
        {
          "AttributeName": "username",
          "KeyType": "HASH"
        },
        {
          "AttributeName": "quest_completed_at",
          "KeyType": "RANGE"
        }
      ],
      "Projection": {
        "ProjectionType": "ALL"
      },
      "ProvisionedThroughput": {
        "ReadCapacityUnits": 1,
        "WriteCapacityUnits": 1
      }
    }
  }
]

Alternatively, you can use a local secondary index to filter completed quests. Local secondary indexes share the same partition key as the main table and increase the partition size (which is limited to 10 GB). They must also be created with the main table and cannot be added or removed afterward.

Global secondary indexes have their own partition and sort keys, and can filter data across the entire table.

You should now be able to log into the AWS Management Console and see the tables and indexes you have created. You can also add data to the tables and query the data using the console.

Conditional Expressions

With our access patterns modeled, let's take a character and give them more coffee, if they have less than a certain amount of coffee in their inventory. We can use a conditional expression to avoid overwriting data and ensure that we are reading the most recent data.

// conditional-expression.ts

import { DynamoDBClient, UpdateItemCommand } from "@aws-sdk/client-dynamodb";
import { marshall } from "@aws-sdk/util-dynamodb";

const client = new DynamoDBClient();

const moreCoffee = async (username: string): Promise<void> => {
  await client.send(
    new UpdateItemCommand({
      TableName: "characters",
      Key: marshall({ username }),
      UpdateExpression: "SET inventory.Coffee = inventory.Coffee + :moreCoffee",
      ConditionExpression: "inventory.Coffee < :maxCoffee",
      ExpressionAttributeValues: {
        ":maxCoffee": { N: "200000" },
        ":moreCoffee": { N: "10000" },
      },
    })
  );
};

void moreCoffee("beautifulcoder");

Keep in mind that all writes must flow through the primary key in the table. Update expressions can tap into the latest data available and perform conditional writes.

DynamoDB has eventual consistency, so it is important to use conditional expressions to avoid overwriting data via a read race condition. This technique will also help keep latencies low and reduce the number of round-trips to the database.

Wrapping Up

In this post, we've seen that when working with DynamoDB, it is important to model your data based on your access patterns. The goal should be to reduce the number of round-trips to the database and keep latencies low.

DynamoDB is a powerful NoSQL database that can store any form of data and scale to any size. Optimizing for access patterns will help you get the most out of DynamoDB.

Happy scaling!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.