GraphQL pagination with DynamoDB - Putting it together

Assuming you have a good understanding of relay pagination and DynamoDB pagination, here's a rundown on how to get the two working in harmony 🥂.

🐙 Creating a resolver

For the majority of this section, it's fair to assume that we're working inside of a resolver such as the following.

const usersResolver = () => async (
  root,
  { first, after, last, before },
) => {
  // ...
};

Determining direction

Before querying the database, we first need to know which direction is being requested by the user.

const isForward = Boolean(first);

The easiest way to do this is to see if the provided first argument has a value. If so, we're working with forward pagination.

Note: Bi-directional pagination within a single query is allowed but is not advised.

Querying the Database

For the query, most of the arguments passed through are going to be your bog-standard query; but there will be some additional attributes we need to pass through.

ScanIndexForward needs to be passed a boolean dependent on the direction of the query (i.e. isForward from the previous example).

ExclusiveStartKey is going to be the client provided cursor (i.e. before or after arguments). The current SDK doesn't support the value null so make sure to fall back to undefined for cases where a cursor isn't present.

await documentClient
  .query({
    ScanIndexForward: isForward,
    ExclusiveStartKey: before || after || undefined,
    // The rest of your query
  })
  .promise();

Page sizing

A single query won't be enough to guarantee that the client provided page size is satisfied. In order to work around this, we're going to need to create a utility to iterate through one or more DynamoDB pages to populate our collection.

export const paginateQuery = <R>(client: DynamoDB.DocumentClient) => async <T = R>({
  params,
  pageSize,
  acc = [],
}: {
  params: DynamoDB.DocumentClient.QueryInput;
  pageSize: number;
  acc?: T[];
}): Promise<{ page: T[]; hasNextPage: boolean }> => {
  const remaining = pageSize - acc.length;
  const result = await client.query(params).promise();
  const newItems = result.Items || [];
  const newAcc = [...acc, ...(newItems.slice(0, remaining) as T[])];

  // Query exhausted
  if (!result.LastEvaluatedKey) {
    return {
      page: newAcc,
      hasNextPage: newItems.length > remaining,
    };
  }

  if (
    // Page needs to be filled more
    newAcc.length < pageSize ||
    // page full but hasNextPage unknown
    newItems.length <= remaining
  ) {
    return paginateQuery(client)({
      params: {
        ...params,
        ExclusiveStartKey: result.LastEvaluatedKey,
      },
      pageSize,
      acc: newAcc,
    });
  }

  return {
    page: newAcc,
    hasNextPage: true,
  };
};

Having pieced this together, the previous DynamoDB query can now instead call this utility and the requested page size is passed along.

const { page, hasNextPage } = await paginateQuery(documentClient)({
  pageSize: first || last,
  params: {
    ScanIndexForward: isForward,
    ExclusiveStartKey: before || after || undefined,
    // The rest of your query
  },
});

Constructing Edges

The response from DynamoDB equates to the nodes in our response Edges. A cursor is also required to be colocated with these nodes.

In this example, the query is on a table (rather than an index) so the keys required correspond to the partition key and sort key of the table.

For index queries, see the Cursor Construction section of the DynamoDB Pagination post.

const cursorKeys = ['id', 'dateOfBirth'] as const;
const edges =  page.map((node) => ({
  node,
  cursor: cursorKeys.reduce((agg, key) => ({ ...agg, [key]: node[key] }), {}),
}));

Correcting edge order

While DynamoDB inverts sort-order when paginating backward, Relay does not. For this reason, the order of edges needs to be reversed if backward pagination is being used.

if (!isForward) {
  edges.reverse();
}

Constructing PageInfo

The task is almost complete! The final part to this pagination saga is to put together the PageInfo response.

Cursors

Assuming the edges have already been ordered correctly (see above) the start and end cursors can easily be set by getting the cursor values of the first and last edges.

const pageInfo = {
  startCursor: edges[0]?.cursor,
  endCursor: edges[edges.length - 1]?.cursor,
  // ...
};

Next pages

Assuming the client is stateful, there's little-to-no need to tell the client whether there is a page available in the opposite direction. For this purpose, we can default to false for hasPreviousPage and hasNextPage for forward and backward pagination, respectively.

const pageInfo = {
  // ...
  ...(isForward
    ? { hasNextPage, hasPreviousPage: false }
    : { hasNextPage: false, hasPreviousPage: hasNextPage }),
};

Final result

Here's what our resolver looks like after putting all these parts together.

const usersResolver = () => async (root, { first, after, last, before }) => {
  const isForward = Boolean(first);
  const { page, hasNextPage } = await paginateQuery(documentClient)({
    pageSize: first || last,
    params: {
      ScanIndexForward: isForward,
      ExclusiveStartKey: before || after || undefined,
      // ...
    },
  });

  const cursorKeys = ["id", "dateOfBirth"] as const;
  const edges = page.map((node) => ({
    node,
    cursor: cursorKeys.reduce((agg, key) => ({ ...agg, [key]: node[key] }), {}),
  }));

  if (!isForward) {
    edges.reverse();
  }

  const pageInfo = {
    startCursor: edges[0]?.cursor,
    endCursor: edges[edges.length - 1]?.cursor,
    ...(isForward
      ? { hasNextPage, hasPreviousPage: false }
      : { hasNextPage: false, hasPreviousPage: hasNextPage }),
  };

  return { edges, pageInfo };
};

🚀 Conclusion

If you've gotten this far - congratulations! You are now a pagination expert™ and are ready to take this on in the real world 🌍!

For the sake of keeping this concise, I've left out a few additional steps (namely optimising cursors and input validation). If you would like to see that follow up post, be sure to let me know 💬.

---

Thanks for reading!

If you enjoyed this post, be sure to react 🦄 or drop a comment below with any thoughts 🤔.

You can also hit me up on twitter - @andyrichardsonn

Disclaimer: All thoughts and opinions expressed in this article are my own.

Comments

[Pedro Arantes]

Hey @andyrichardsonn ! First, congratulations, I really appreciate the series, the way you wrote it, and the GIFs :)

Cursor-based pagination with DynamoDB is a topic I have an interest in because I work with it every day. I'd like to know if you have plans to create an NPM package for this method. I've created the dynamodb-cursor-based-pagination and I'm going to use some ideas of your code, like the exhaustQuery and passing documentClient as a parameter.

Tell me if you have plans to transform this into an NPM package, maybe we can work together.

[Andy Richardson]

Hey Pedro, thanks for the comment!

Feel free to use all/any of the code examples 👍 The project you have looks interesting and I'll definitely keep an eye on it!

I think one of the larger challenges with encapsulating pagination into a library is how much variance needs to be considered.

For example, there is some required domain/schema knowledge to be able to construct cursors, users might require (or want to avoid) calculating the hasPreviousPage value, and I'm sure there are a large number of additional edge cases (post-fetch filtering, etc).

It looks like you've already made a tonne of headway so maybe this is/could be a solved problem for those who have less exotic needs and are happy working within the constraints of a library 🚀