Intro
Have you ever needed to read data and wondered how to paginate it effectively in DynamoDB? In this article, we'll tackle this challenge using a Fastify plugin that leverages the AWS SDK. Let's dive in!
One step back: how DynamoDB works
DynamoDB is like a giant electronic filing cabinet for your data. Imagine it as a collection of drawers (partitions), each holding folders (items) with pieces of information.
Partition Key: Think of the partition key as the label on each drawer. It helps DynamoDB decide where to put and find your data. Every item must have a unique partition key. For example, if you're storing information about people, the partition key might be their unique ID.
Sort Key: If you want to organize the folders within a drawer, you use a sort key. It's like arranging the folders by date or name. It helps you quickly find what you need inside a partition.
Query vs. Scan: Imagine you're looking for specific folders. A query is like searching for folders with a specific label in a single drawer. It's super quick because DynamoDB knows exactly where to look. On the other hand, a scan is like going through all the folders in a drawer, looking for what you need. It can be slower and less efficient because it checks everything.
In a nutshell, DynamoDB stores your data in drawers (partitions), uses labels (partition keys) to find them quickly, and can sort them using additional labels (sort keys). You can either search for specific items (query) or look through everything (scan) depending on your needs.
Example: DynamoDB Pagination with Fastify Plugin
In this example, we'll explore how to paginate data in DynamoDB using a Fastify plugin. We have a DynamoDB table called "test-pager" with the following structure:
- userId: This is our partition key, a unique identifier for each user (string).
- deviceId: The sort key, which helps us organize data for each user's device (string).
Each item in the table also has an attribute:
- address: A string attribute that holds the device address.
Understanding the Table
To better grasp this setup, think of "test-pager" as a collection of user devices. The partition key, "userId," separates users, and the sort key, "deviceId," allows us to associate multiple devices with a user. For instance:
-
Item 1
- userId: "u01"
- deviceId: "d01"
- address: "10.0.1.1"
-
Item 2
- userId: "u01"
- deviceId: "d02"
- address: "10.0.2.1"
Limit and "Offset"
Pagination in DynamoDB using the AWS SDK for a Query operation is like flipping through a book. When you use 'Limit,' you're telling DynamoDB how many results you want to see on each page, like asking for 10 pages at a time.
'ExclusiveStartKey' is like a bookmark that remembers where you stopped reading. It helps you start from the right place on the next page. It's different from an offset because you don't count items; you rely on DynamoDB to remember.
'Limit' controls how many items per page, and 'ExclusiveStartKey' remembers where to start on the next page. This way, you can efficiently paginate through large datasets without the need to manually count or skip items.
Fastify App
In our Fastify app, we've set up an API endpoint at '/api/devices' for reading user devices. Users can provide a 'userId,' optionally set a 'limit' (defaulting to 10), and include an optional 'startKey' to resume pagination from where they left off.
route/list.ts
export default async function (
fastify: FastifyInstance,
_opts: FastifyPluginOptions,
): Promise<void> {
fastify.get<{ Querystring: DeviceRequestDtoType, Reply: DynamoDBPaginatedResult<DeviceDtoType> }>(
'/',
{
schema: {
querystring: DeviceRequestDto
}
},
async (request, reply) => {
try {
const { userId, limit, startKey } = request.query
const result = await fastify.listDevices(userId, limit, startKey)
return reply.send(result)
} catch (error) {
request.log.error(error)
return reply.code(500).send()
}
},
)
}
The feature 'listDevices', registered as a plugin, is invoked within the route:
features/devices.list.ts
async function listDevicesPlugin(
fastify: FastifyInstance,
_opts: FastifyPluginOptions,
): Promise<void> {
const listDevices = async (
userId: string,
limit: number,
startKey?: string
): Promise<DynamoDBPaginatedResult<DeviceDtoType>> => {
const queryCommand = new QueryCommand({
TableName: 'test-pager',
KeyConditionExpression: 'userId = :userId',
ExpressionAttributeValues: { ':userId': { S: userId } },
Limit: limit,
ExclusiveStartKey: startKey ? fastify.DynamoDBKeyUtil.decode(startKey) : undefined,
})
const deviceTransformer = (item: Record<string, AttributeValue>): DeviceDtoType => ({
userId: item.userId.S as string,
deviceId: item.deviceId.S as string,
address: item.address.S as string,
})
const pager = fastify.DynamoDBPager<DeviceDtoType>(queryCommand, deviceTransformer);
return await pager()
}
fastify.decorate('listDevices', listDevices)
}
export default fp(listDevicesPlugin)
plugins/dynamodb.pager.ts
async function dynamoDBPagerPlugin(
fastify: FastifyInstance,
_opts: FastifyPluginOptions,
): Promise<void> {
const DynamoDBPager = <T>(
queryCommand: QueryCommand,
itemTransformer: (item: Record<string, AttributeValue>) => T
) => async (): Promise<DynamoDBPaginatedResult<T>> => {
const queryResult = await fastify.dynamoDB.send(queryCommand)
const items = queryResult.Items?.map(itemTransformer) ?? []
const paginatedResult: DynamoDBPaginatedResult<T> = {
items
}
if (queryResult?.LastEvaluatedKey) {
paginatedResult.lastKey = fastify.DynamoDBKeyUtil.encode(queryResult.LastEvaluatedKey)
}
return paginatedResult
}
fastify.decorate('DynamoDBPager', DynamoDBPager)
}
export default fp(dynamoDBPagerPlugin)
Encoding and Decoding startKey and lastKey
When working with DynamoDB, you'll notice that unlike the numeric 'offset' parameter used in other databases like MySQL, DynamoDB uses 'LastEvaluatedKey' and 'ExclusiveStartKey' as objects to manage pagination. To work with these keys, you'll need encoding and decoding methods (another Fastify Plugin), which we'll illustrate with some code examples shortly:
plugins/dynamodb.keyutil.ts
async function dynamoDBkeyUtilPlugin(
fastify: FastifyInstance,
_opts: FastifyPluginOptions,
): Promise<void> {
const DynamoDBKeyUtil = {
encode: (key: Record<string, AttributeValue>): string => btoa(JSON.stringify(key)),
decode: (key: string): Record<string, AttributeValue> => JSON.parse(atob(key))
}
fastify.decorate('DynamoDBKeyUtil', DynamoDBKeyUtil)
}
export default fp(dynamoDBkeyUtilPlugin)
Conclusions
DynamoDB, Fastify's plugin ecosystem, and TypeScript are powerful tools in your development arsenal. By understanding DynamoDB's nuances, leveraging Fastify's modularity, and embracing TypeScript's benefits, you can craft applications that are not only efficient but also robust and easily maintainable for the long haul.
Three interesting points:
Understanding DynamoDB: To make the most of DynamoDB, it's essential to grasp its unique characteristics. By working with its distributed data model and efficient indexing, you can harness its full potential for data storage and retrieval.
Fastify's Modular Ecosystem: Fastify's plugin-rich environment is a game-changer for building applications. With modular components readily available, development becomes more agile and efficient, allowing you to assemble your app like building blocks.
The Power of TypeScript: TypeScript plays a vital role in creating robust and maintainable applications. Its static typing helps catch errors early in the development process, making your codebase more reliable and easier to maintain over time.
Top comments (0)