DEV Community: Rich

How to deploy Remix apps with SSR to AWS Amplify

Rich — Tue, 02 Apr 2024 15:00:21 +0000

The AWS Amplify team recently announced the Amplify Hosting deployment specification. This is a way to deploy applications to AWS Amplify with server side rendering (SSR) support. While there are guides or support for many frameworks including Astro, NextJS and Nuxt I couldn't find one for Remix.

As I'm about to start working on a Shopify app (Shopify provides templates for Remix) and I didn't want to switch infrastructure providers I decided to investigate using how difficult it would be to use Amplify to host a Remix app with SSR.

The first step is to create a build file named amplify.yml which Amplify uses to build the project. I was deploying a new project and Amplify detected the file automatically during the initial deployment. Our build script handles four important tasks:

Runs npm run build to build the application. Remix puts the output in into the build folder.
Moves and restuctures the build folder output to meet the Amplify hosting specification by
Reduce the size of our node-modules folder by running npm ci --omit dev to create a node-modules that only includes production dependencies. This is important for larger project as there is a limit on the maximum size of this folder. It also helps reduce cold start times. The folder is moved into the compute/default folder so the modules are available at runtime.
Finally there are two files that we will create shortly that need to be copied into place.

The complete amplify.yml is below:

version: 1baseDirectory: .amplify-hostingfrontend: phases: preBuild: commands: - npm ci build: commands: - npm run build - mv build .amplify-hosting - mv .amplify-hosting/client .amplify-hosting/static - mkdir -p .amplify-hosting/compute - mv .amplify-hosting/server .amplify-hosting/compute/default - npm ci --omit dev - cp package.json .amplify-hosting/compute/default - cp -r node_modules .amplify-hosting/compute/default - cp server.js .amplify-hosting/compute/default - cp deploy-manifest.json .amplify-hosting/deploy-manifest.json artifacts: files: - "**/*" baseDirectory: .amplify-hosting

AWS Amplify will now package and deploy our .amplify-hosting folder.

The deploy-mainfest.json has two main tasks:

Tell Amplify how to route traffic
Tell Amplify how to configure and start the compute resources

This routing should not be confused with rewriting and redirecting traffic. It's purpose is to indicate if traffic should be handled as static or compute. Here I'm using the less than perfect approach that files with a . should be treated as static first and fall back to compute if they don't exist. Everything else should be treated as compute. This means that static files must have a . in the filename.

For the compute configuration I've set the runtime to Node 20 and the project should be started by using node server.js

The full deploly-mainfest.json is:

{ "version": 1, "framework": { "name": "remix", "version": "2.8.1" }, "routes": [{ "path": "/*.*", "target": { "kind": "Static", "cacheControl": "public, max-age=2" }, "fallback": { "kind": "Compute", "src": "default" } }, { "path": "/*", "target": { "kind": "Compute", "src": "default" } }], "computeResources": [{ "name": "default", "runtime": "nodejs20.x", "entrypoint": "server.js" }]}

Finally I need to create a small Javascript file called server.js. This file launches an Express server on port 3000 which listens for requests and passes them to Remix. Remember to add express as a production dependency so this will work.

npm i express --save

The complete server.js is:

import remix from "@remix-run/express";import express from "express";import * as build from "./index.js";const app = express();const port = 3000;app.all( "*", remix.createRequestHandler({ build, }));app.listen(port, () => { console.log(`Example app listening on port ${port}`)})

That's it! You should be able to deploy the project to AWS Amplify and when you go to the URL provided your request will be execute on the server.

Moving "server-only" to the top of the imports

Rich — Mon, 01 Apr 2024 15:00:45 +0000

Like many developers I use Prettier to keep my code base looking consistent. One of the plugins we use is @trivago/prettier-plugin-sort-imports which automatically sorts the order of our imports.

While converting an old Gatsby website to NextJS we wanted to use the server-only package to prevent leaking secrets. By positioning it at the top of the file it would also allow us to quickly identify server only code. Of course the sort imports plugin had very different ideas and diligently moved it further down the list.

import { type ClientPerspective, type QueryParams, createClient } from "next-sanity";import "server-only";

Fixing this should just require adding ^server-only$ to the start of the importOrder setting but the plugin keeping putting it last.

{ "plugins": ["@trivago/prettier-plugin-sort-imports"], "importOrder": ["^server-only$", "^[./]"], "importOrderSeparation": true, "importOrderSortSpecifiers": true}

After a little mucking around I realised that it was sorting the modules correctly and putting server-only first but the sorted import list always appeared after the third party modules. This made it look like the plugin wasn't working. Luckily fixing this was trivial thanks to the <THIRD_PARTY_MODULES> placeholder that could be used to sort where they appeared.

{ "plugins": ["@trivago/prettier-plugin-sort-imports"], "importOrder": ["^server-only$", "<THIRD_PARTY_MODULES>", "^[./]"], "importOrderSeparation": true, "importOrderSortSpecifiers": true}

With done that all my server-only imports moved to the top of their files.

import "server-only";import { type ClientPerspective, type QueryParams, createClient } from "next-sanity";

Invalid Grant error with AWS IAM Identity Center

Rich — Fri, 29 Mar 2024 09:00:00 +0000

Recently I was setting up a new computer which involved configuring the AWS CLI to use IAM Identity Center (formerly AWS SSO) to access my accounts. Normally this is a prety straight forward proposition. After running aws configure sso command you need to provide four pieces of information:

Session Name
Start URL
Region
Registration Scopes

AWS then authenticates you, you select your account, answer some more questions and it's done.

This time I keep getting an invalid_grant error after I authenticated myself.

The problem and solution turned out to be really simple. I selected the wrong region for IAM Identity Center. In my defence I mostly work with IAM Identity Center in my closest region but this was an older account and it was setup in a different region. Once I had the correct region everything worked correctly.

Solve the AppSync Lambda Resolver N+1 Problem with BatchInvoke

Rich — Tue, 02 Jan 2024 13:00:00 +0000

If you're not familiar with the GraphQL N+1 problem then consider the following query.

query { authors { id name books { id name } }}

The "authors" resolver is executed once and returns a list of N authors. The "books" resolver is then executed once for each author. This is the N+1 problem because you execute N resolvers for the books plus 1 for the authors.

As your query becomes deeper the number of resolvers executed increases significantly. Let's assume there are 10 authors, each with 5 books. To fetch the authors and books there are 11 resolvers executed but if we also request publisher information for each book we now execute 61 resolvers (1 + 10 + 10 x 5).

query { authors { id name books { id name publisher { id name } } }}

When configuring AppSync to use a Lambda function as a resolver you can use the Invoke or BatchInvoke operation.

The Invoke operation will cause AppSync to invoke your Lambda function every time it needs to execute the resolver. For the "authors" resolver in our example that is acceptable because it is only executed once.

import { AppSyncResolverHandler } from 'aws-lambda';export const authorsHandler: AppSyncResolverHandler<unknown, { id: string; name: string }[]> = async (event) => { const authors = [{ id: '1', name: 'Tajeddigt Olufemi' }, { id: '2', name: 'Lelio Miodrag' }, { id: '3', name: 'Aineias Vladimir' }, { id: '4', name: 'Sachin Lamya' }, { id: '5', name: 'Kamakshi Cosme' }, { id: '6', name: 'James Sharmila' }, { id: '7', name: 'Holden Wulffld' }, { id: '8', name: 'Quidel Bahdan' }, { id: '9', name: 'Reinout Johanna' }, { id: '10', name: 'Til Nikica' }, { id: '11', name: 'Metodj Maxima' }, { id: '12', name: 'nh Ester' }, { id: '13', name: 'Mxim Kristina' }, { id: '14', name: 'Yedidia Jafar' }, { id: '15', name: 'Lone Mariusz' }, { id: '16', name: 'Vitya Franjo' }, { id: '17', name: 'Malvolio Lochlann' }, { id: '18', name: 'Evette Dierk' }, { id: '19', name: 'Nnenna Basileios' }, { id: '20', name: 'Dmitrei Iya' },]; return authors;};

For the "books" resolver we really want to use the BatchInvoke operation. To use it with the direct Lambda resolver integration add the MaxBatchSize property to your resolver definition.

  BooksResolver: Type: AWS::AppSync::Resolver Properties: ApiId: !GetAtt Api.ApiId DataSourceName: !GetAtt BooksDataSource.Name TypeName: Author FieldName: books MaxBatchSize: 1000

AppSync will now invoke your Lambda with an array of up to MatchBatchSize events. This allows your Lambda to process multiple resolvers at once instead of individually.

import { AppSyncBatchResolverHandler } from 'aws-lambda';export const lambdaHandler: AppSyncBatchResolverHandler< unknown, { id: string; name: string }[], { id: string; name: string }> = async (events) => { return events.map((event) => { const books: { id: string; name: string }[] = []; for (let i = 1; i <= 20; i++) { books.push({ id: `${event.source.id}-${i}`, name: `Book ${i}` }); } return books; });};

By batch handling resolver executions in a single Lambda invocation you can improve performance by reducing the number of cold starts and lower costs through invoking less Lambda functions.

Tip: In production you may want to experiment with values forMaxBatchSizeas you are trading off the risk of a cold start and cost versus being able to process requests in parallel.

What about our publishers?

You should always use BatchInvoke for nested Lambda resolvers when possible. At each level of nesting AppSync will determine the events that need to be sent to your Lambda resolver then pass them in array of MaxBatchSize length resulting in the minimum number of Lambda functions being executed. With a sufficiently high enough MaxBatchSize value you may only need to invoke one Lambda function for each level of nesting in your query.

By using BatchInvoke you may also be able to optimize your resolvers through more efficient processing of requests.

In our example we are requesting information about the publisher for each book. It's likely that publishers will be repeated many times. If we used Invoke then each resolver execution would use GetItem to fetch the published record from DynamoDB. This means the same publisher record would be retrieved multiple times. With BatchInvoke we can filter a list of unique publisher ID's, use BatchGetItem to fetch them all at once from DynamoDB then map over the events to return the correct publish record for each resolver. This reduces our usage of DynamoDB and makes it quicker.

Sprint Timeline

Rich — Wed, 17 Mar 2021 00:00:00 +0000

If you need to run an agile sprint timeline this document might help.

Timeline

The timeline for a 2 week sprint

Day	Day of Week	Description
-5	Wednesday	Sprint Planning Meeting
-4	Thursday
-3	Friday
-2	Saturday
-1	Sunday
0	Monday	Sprint Grooming / Sprint Start
1	Tuesday	Standup
2	Wednesday	Standup
3	Thursday	Standup
4	Friday	Standup
5	Saturday
6	Sunday
7	Monday	Standup
8	Tuesday	Standup
9	Wednesday	Standup
10	Thursday	Standup
11	Friday	Standup
12	Monday	Final Day / Standup / Retro

Points

The number of points a developer should target during the sprint depends on the number of days they are working.

Days Worked	Min Points	Max Points
1	1	2
2	3	4
3	5	6
4	7	8
5	9	10
6	11	12
7	13	14
8	14	16
9	16	18
10	18	20

When allocating points to a task use the following table. If a story takes longer than 5 days it should be broken down into multiple stories.

Points	Days
1	½
2	1
3	1-2
5	2-3
8	5

Install nvm and Node.js for macOS 11

Rich — Mon, 15 Mar 2021 00:00:00 +0000

The Node Version Manager is an easy to way to run multiple versions of Node. Having upgraded to macOS 11 recently I found that I had accidentally installed a Node using Homebrew and I needed to get nvm working again.

The process was fairly simple.

Uninstall the Homebrew Node:

brew uninstall node

Try to install the Homebrew nvm:

brew install nvm

Ok, this didn't work so well:

######################################################################## 100.0%
Error: Your Command Line Tools (CLT) does not support macOS 11.
It is either outdated or was modified.
Please update your Command Line Tools (CLT) or delete it if no updates are available.
Update them from Software Update in System Preferences or run:
  softwareupdate --all --install --force

If that doesn't show you any updates, run:
  sudo rm -rf /Library/Developer/CommandLineTools
  sudo xcode-select --install

Alternatively, manually download them from:
  https://developer.apple.com/download/more/.

Error: An exception occurred within a child process:
  SystemExit: exit

I knew I was running the latest version of everything so I jumped straight to:

sudo rm -rf /Library/Developer/CommandLineTools
sudo xcode-select --install

With that done I could finally install nvm:

brew install nvm

I then cleaned out my old .nvm folder and created a new one:

rm -rf ~/.nvm && mkdir ~/.nvm

As I switched from bash to zsh when upgrading to macOS 11 I had to add some lines to my ~/.zshrc:

export NVM_DIR="$HOME/.nvm"
[-s "/usr/local/opt/nvm/nvm.sh"] && . "/usr/local/opt/nvm/nvm.sh" # This loads nvm
[-s "/usr/local/opt/nvm/etc/bash_completion.d/nvm"] && . "/usr/local/opt/nvm/etc/bash_completion.d/nvm" # This loads nvm bash_completion

Before continuing I ran the source command but you could simply restart your shell:

source ~/.zshrc

It was now time to install the version of Node I required:

nvm install v14.16.0

Finally, check that Node is installed correctly.

node --version

AppSync Resolver Optimization by Removing Unnecessary GetItem's

Rich — Sun, 16 Aug 2020 00:00:00 +0000

I had a front end developer comment that one of API's was slow. The developer was trying download all of the products and their related image. After a quick investigation it turned out we had hit the dreaded N+1 problem.

Using AppSync's new context info object we've been able to apply a simple optimization that has significantly reduced the number of DynamoDB queries our API was making.

A simplified version of the schema would look something like this:

type Image {
  id: ID!
  url: String
}

type Product {
  id: ID!
  name: String
  image: Image
}

type Query {
  listImages: [Image]
  listProducts: [Products]
}

The AppSync resolvers for Query.listProducts and Query.listImages both performed a DynamoDB Query allowing one request to return multiple records. The Product.image resolver used a GetItem to fetch the image for each product.

In my head the developer would use a query like this

query ListProductsWithImage {
  listProducts {
    id
    name
    image {
      id
      url
    }
  }
}

You might notice a problem here. While listProducts can be performed with a single DynamoDB operation I'm performing another DynamoDB operation for every record that it returns. The N+1 problem.

It turned out that the developer was actually using two queries and joining the data in the application. The first downloaded all of the images

query ListImages {
  listImages {
    id
    url
  }
}

The second all of the products

query ListProducts {
  listProducts {
    id
    name
    image {
      id
    }
  }
}

While this could have been efficient it wasn't because the Product.image resolver was still performing a GetItem for each product returned by Query.listProducts.

This was crazy because we store the Image id in the imageId attribute as part of the Product record in DynamoDB.

One solution would be to expose the imageId in Product but this is discouraged in GraphQL schema design because it exposes the internal implementation details to an external client making it harder to change the implementation later.

With the new $ctx.info object and early returning I can now optimize my resolvers for this use case by adding the following code at the top of the Product.image request resolver.

#if ($ctx.info.selectionSetList.size() == 1 && $ctx.info.selectionSetList[0] == "id")
  #return({ "id": "$ctx.source.imageId" })
#end

If id is the only image field requested it will now return early, before the GetItem operation, using $ctx.source.imageId as the id.

This small optimization improves API performance and saves money by removing unnecessary GetItem requests without exposing the internal implementation but still allows the client to fetch additional fields if required.

Disabling Cognito User Pools Authentication for a Single Mutation with AppSync

Rich — Thu, 28 May 2020 00:00:00 +0000

Typically I combined AppSync with Cognito User Pools for authorization. This works great for API's where the user is logged in but what if you need a few queries or mutations to work for users who aren't logged in?

A common example of this is a mutation that allows people to register.

While AppSync doesn't allow unauthenticated requests you can use API key authorization to get around the need for a user to be logged in.

Start by setting up AppSync with Cognito User Pools as the default authorization mode and API key as an additional authorization provider. This will protect the API using Cognito User Pools authorization but allow us to enable API key authorization for some fields.

Next I need a schema with a mutation.

type Account {
  id: ID!
  email: AWSEmail!
  name: String!
}

input RegisterAccountInput {
  email: AWSEmail!
  name: String!
  password: String!
  ## Other registration fields
}

type Query {
  findAccount(id: ID!): Account
}

type Mutation {
  registerAccount(input: RegisterAccountInput!): Account
}

By default the entire schema can only be accessed if the request uses Cognito User Pools authorization. By adding @aws_api_key @aws_cognito_user_pools to a type/field you can allow both authorization methods. If you only want to allow API key then you can use @aws_api_key.

In the following example I've limited access to the registerAccount mutation to only requests using API key authorization and you can only access the id field on the Account that it returns.

type Account {
  id: ID! @aws_api_key @aws_cognito_user_pools
  email: AWSEmail!
  name: String!
}

input RegisterAccountInput {
  email: AWSEmail!
  name: String!
  password: String!
  ## Other registration fields
}

type Query {
  findAccount(id: ID!): Account
}

type Mutation {
  registerAccount(input: RegisterAccountInput!): Account @aws_api_key
}

For the user to access the other Account fields they would need to switch to Cognito User Pools authorization and use findAccount.

How to build an AppSync API using a single table DynamoDB design

Rich — Fri, 15 May 2020 00:00:00 +0000

If you want to start a flame war in the AWS AppSync community then ask if you should use a single or multiple table design for DynamoDB. There's no shortage of opinions in both directions 🔥🔥🔥

In this article I'm going to show you how I built a generic single table DynamoDB solution for AWS AppSync.

Historical Context

Before I explain "how" it's important to remember that DynamoDB did not support on-demand capacity (pay per use pricing) when AppSync was originally released. Managing the read and write capacity across multiple tables was a lot of extra work. Under provisioning could result in your API failing and over provisioning was expensive.

By using a single table I could reduce the amount of time spent managing table capacity and save a lot of money. That is why began exploring how to build an AppSync API using only a single DynamoDB table.

Primary Key

For the tables primary key I used a generic attribute named PK that does not appear in the GraphQL schema. Keeping the primary key for the table out of the GraphQL schema allows me to use any field in the GraphQL type as the primary ID for a record.

Note: To keep my examples simple I will use id as the primary field but you can use a different field or fields.

Mapping the primary ID for the GraphQL type to PK is handled in the AppSync resolver request template. There are two patterns I recommend when mapping fields to PK.

NODE#${id} if you have globally unique ID's
${__typename}#${id} if you do not have globally unique ID's

Example: Let's start with a simple GraphQL schema for farms.

type Farm {
  id: ID!
  name: String
}

input CreateFarmInput {
  name: String!
}

type Mutation {
  createFarm(input: CreateFarmInput!): Farm
}

type Query {
  findFarm(id: ID!): Farm
}

The request resolver template for createFarm would be

#set( $id = $util.autoId() )
$util.qr($ctx.args.input.put("id", $id))
$util.qr($ctx.args.input.put("__typename", "Farm"))

{
  "version" : "2017-02-28",
  "operation" : "PutItem",
  "key" : {
    "PK": $util.dynamodb.toDynamoDBJson("NODE#$id"),
  },
  "attributeValues" : $util.dynamodb.toMapValuesJson($ctx.args.input)
}

This template generates an id for the new farm that is added to the input along with the __typename. It then prefixes that ID with NODE# to generate the PK value for the DynamoDB key.

When invoked with the following input

mutation {
  createFarm(input: { name: "Old MacDonald's" }) {
    id
  }
}

It will create a record in DynamoDB similar to

PK	__typename	id	name
NODE#1234	Farm	1234	Old MacDonald's

The farm can then be retrieved using findFarm that accepts an id which is mapped to the PK attribute in the request template:

{
  "version": "2017-02-28",
  "operation": "GetItem",
  "key": {
    "PK": $util.dynamodb.toDynamoDBJson("NODE#$ctx.args.id"),
  }
}

SECURITY WARNING

When using the NODE#${id} pattern you must check the __typename is one that you expect in every resolver that fetches, updates or deletes records. Failure to do this may allow someone to by pass security by using the ID from one type with the resolver for a different type that is more permissive.

The examples in this article do not include this check because I am focusing on the DynamoDB access pattern and not AppSync access controls.

Unique Secondary Fields

Requiring additional fields in our GraphQL type to be unique is a common problem. The Farm type contains an id field that uniquely identifies a record and never changes. It also has a name field that can change but must be unique.

With DynamoDB only the primary key on the table is unique. We can't add this constraint to other attributes. To solve this I can use TransactionWriteItems to write a record under different keys. The first key is the NODE#${id} as previously discussed. The second key uses the pattern ${__typename}#${fieldname}#${value}.

The createFarm resolver request template will now look like

#set( $id = $util.autoId() )
$util.qr($ctx.args.input.put("id", $id))
$util.qr($ctx.args.input.put("__typename", "Farm"))

{
  "version": "2018-05-29",
  "operation": "TransactWriteItems",
  "transactItems": [
    {
      "table": "YOUR-TABLE-NAME",
      "operation": "PutItem",
      "key": {
        "PK": $util.dynamodb.toDynamoDBJson("NODE#$id")
      },
      "attributeValues": $util.dynamodb.toMapValuesJson($ctx.args.input),
      "condition": {
        "expression": "attribute_not_exists(PK)",
        "returnValuesOnConditionCheckFailure": false
      }
    },
    {
      "table": "YOUR-TABLE-NAME",
      "operation": "PutItem",
      "key": {
        "PK": $util.dynamodb.toDynamoDBJson("Farm#name#$ctx.args.input.name.toLowerCase()")
      },
      "attributeValues": $util.dynamodb.toMapValuesJson($ctx.args.input),
      "condition": {
        "expression": "attribute_not_exists(PK)",
        "returnValuesOnConditionCheckFailure": false
      }
    },
  ]
}

Because I need to use the 2018-05-29 resolver template version for TransactionWriteItems and I only want to return one record my resolver response template is a little more complicated

#if ( $ctx.error )
  $util.error($ctx.error.message, $ctx.error.type)
#end
$util.toJson($ctx.args.input)

The main thing to understand is that I'm returning the $ctx.args.input instead of $ctx.result. This contains our Farm type with the id and __typename set correctly by the request template.

In DynamoDB the records will be stored as:

PK	__typename	id	name
NODE#1234	Farm	1234	Old MacDonald's
Farm#name#old macdonald's	Farm	1234	Old MacDonald's

If I call the createFarm mutation twice with the same name it will now fail because a record with the key Farm#name#old macdonald's already exists.

You may have noticed that I converted the name to lower case before using it in the PK attribute for the second record. This prevents people from getting around the unique name restriction by using different combinations of upper and lower case letters. The name attribute itself still retains the original case and that's the value the user will see when the record is queried via GraphQL.

As a side benefit I now have a fast way to retrieve a Farm record from the name using GetItem. This is useful if you have a natural key like email address or ISBN that you want to fetch records by.

One to One Relationships

One to One relationships are simple to implement. Let's add a logo to the Farm that returns an Image type.

type Image {
  id: ID!
  url: String
}

type Farm {
  id: ID!
  name: String
  logo: Image
}

I will also add a logoId field to the input for createFarm

mutation {
  createFarm(input: { name: "Old MacDonald's", logoId: "1111" }) {
    id
  }
}

In DynamoDB this will be saved as

PK	__typename	id	imageId	name	url
NODE#1234	Farm	1234	1111	Old MacDonald's
NODE#1111	Image	1111			logo.gif

When the request resolver template for logo is executed the DynamoDB record for the farm is available in $ctx.source. This allows us to perform a GetItem with the correct PK value to retrieve the Image.

{
  "version": "2017-02-28",
  "operation": "GetItem",
  "key": {
    "PK": $util.dynamodb.toDynamoDBJson("NODE#$ctx.source.logoId"),
  }
}

Traversing both directions in a one to one relationship requires storing the ID for the opposite side in each record. In our example this means storing the imageId with the Farm and farmId with the Image.

Many to One Relationships

Many to One relationships work almost exactly the same as one to one relationships. On the many side you need to store the ID for the one side of the relationship.

Let's expand our farm to include cow's.

type Cow {
  id: ID!
  name: String
  farm: Farm
}

input CreateCowInput {
  name: String!
  farmId: ID!
}

type Farm {
  id: ID!
  name: String
}

input CreateFarmInput {
  name: String!
}

type Mutation {
  createCow(input: CreateCowInput!): Cow
  createFarm(input: CreateFarmInput!): Farm
}

type Query {
  findFarm(id: ID!): Farm
}

To create a new cow we would execute a mutation like

mutation {
  createCow(input: { name: "Bessie", farmId: "5678" }) {
    id
  }
}

The request resolver template for createCow is the same as createFarm with the __typename changed from Farm to Cow.

#set( $id = $util.autoId() )
$util.qr($ctx.args.input.put("id", $id))
$util.qr($ctx.args.input.put("__typename", "Cow"))

{
  "version" : "2017-02-28",
  "operation" : "PutItem",
  "key" : {
    "PK": $util.dynamodb.toDynamoDBJson("NODE#$id"),
  },
  "attributeValues" : $util.dynamodb.toMapValuesJson($ctx.args.input)
}

This will create a DynamoDB record similar to:

PK	__typename	id	farmId	name
NODE#1234	Cow	1234	5678	Bessie
NODE#5678	Farm	5678		Old MacDonald's

Our request resolver for the farm field on the Cow type would be:

{
  "version": "2017-02-28",
  "operation": "GetItem",
  "key": {
    "PK": $util.dynamodb.toDynamoDBJson("NODE#$ctx.source.farmId"),
  }
}

One to Many Relationships

One to Many relationships are more complicated and require writing additional attributes to the DynamoDB record. To implement these I use generic GSI's named GSI01, GSI02, GSI03, etc. Each of these will have a corresponding partition key PK01, PK02, PK03, etc and sort key SK01, SK02, SK03. The additional data for these attributes is added by the resolver request template during the create/update mutation.

In PKxx you typically write the value ${ __typename}#${othersideId} where __ typename is the name of the type you're writing data for (the many side) and othersideId is the ID for the one side. The value for SKxx will depend on how you want sort the results when traversing from the one to the many. I'll explain this further down. All of this should be done in the resolver template so these details are not exposed in the GraphQL schema.

Let's make a tiny change to our farm schema adding a connection between Farm and Cow.

type Farm {
  id: ID!
  name: String
  cows: [Cow]
}

Everything else will remain the same.

When retrieving the cows from the farm I want them sorted by name. To do this I'll copy the name value into SK01. With the extra information our createCow request template now looks like

#set( $id = $util.autoId() )
$util.qr($ctx.args.input.put("id", $id))
$util.qr($ctx.args.input.put("__typename", "Cow"))
$util.qr($ctx.args.input.put("PK01", "Cow#$ctx.args.input.farmId"))
$util.qr($ctx.args.input.put("SK01", $ctx.args.input.name))

{
    "version" : "2017-02-28",
    "operation" : "PutItem",
    "key" : {
        "PK": $util.dynamodb.toDynamoDBJson("NODE#$id"),
    },
    "attributeValues" : $util.dynamodb.toMapValuesJson($ctx.args.input)
}

When create cow is called with

mutation {
  createCow(input: { name: "Bessie", farmId: "5678" }) {
    id
  }
}

The cow DyanmoDB record is now written as:

PK	__typename	id	farmId	name	PK01	SK01
NODE#1234	Cow	1234	5678	Bessie	Farm#5678	Bessie
NODE#5678	Farm	5678		Old MacDonald's

To get from farm to cow we want the cows resolver request template to Query the GS01 index for a PK01 value that is the farms ID prefixed by the return type. Our cows resolver request template is

{
    "version" : "2017-02-28",
    "operation" : "Query",
    "index": "GS01",
    "query" : {
        "expression": "PK01 = :pk",
        "expressionValues" : {
            ":pk" : $util.dynamodb.toDynamoDBJson("Cow#$ctx.source.id")
        }
    }
}

Let's continue expanding the farm by adding chickens.

type Chicken {
  id: ID!
  name: String
  farm: Farm
}

type Cow {
  id: ID!
  name: String
  farm: Farm
}

input CreateChickenInput {
  name: String!
  farmId: ID!
}

input CreateCowInput {
  name: String!
  farmId: ID!
}

type Farm {
  id: ID!
  name: String
  chickens: [Chicken]
  cows: [Cow]
}

type Mutation {
  createChicken(input: CreateChickenInput!): Chicken
  createCow(input: CreateCowInput!): Cow
}

type Query {
  farm(id: ID!): Farm
}

You can copy the resolver request templates from createCow to createChicken and cows to chickens. Just remember to replace Cow with Chicken inside the templates.

What you might have noticed is that I did not need to create a new GSI. By prefixing the farmId with the __typename we are preventing a data collision which allows us to use the same GSI once for each GraphQL type. This slows the growth in GSI's because the number of GSI's for the table is determined by the GraphQL type using the largest number of GSI's.

The ${returnTypename}#${context.source.id} pattern also works when a field returns multiple types. Let's modify the schema by adding Animal which can be a Chicken or Cow.

union Animal = Chicken | Cow

type Chicken {
  id: ID!
  name: String
  farm: Farm
}

type Cow {
  id: ID!
  name: String
  farm: Farm
}

input CreateChickenInput {
  name: String!
  farmId: ID!
}

input CreateCowInput {
  name: String!
  farmId: ID!
}

type Farm {
  id: ID!
  name: String
  chickens: [Chicken]
  cows: [Cow]
  animals: [Animal]
}

type Mutation {
  createChicken(input: CreateChickenInput!): Chicken
  createCow(input: CreateCowInput!): Cow
}

type Query {
  farm(id: ID!): Farm
}

To make this work I need a second GSI's. In PK02 I'll store Animal#${farmId} using the createChicken and createCow resolver request templates. Here's what createCow now looks like

#set( $id = $util.autoId() )
$util.qr($ctx.args.input.put("id", $id))
$util.qr($ctx.args.input.put("__typename", "Cow"))
$util.qr($ctx.args.input.put("PK01", "Cow#$ctx.args.input.farmId"))
$util.qr($ctx.args.input.put("SK01", $ctx.args.input.name))
$util.qr($ctx.args.input.put("PK02", "Animal#$ctx.args.input.farmId"))
$util.qr($ctx.args.input.put("SK02", $ctx.args.input.name))

{
    "version" : "2017-02-28",
    "operation" : "PutItem",
    "key" : {
        "PK": $util.dynamodb.toDynamoDBJson("NODE#$id"),
    },
    "attributeValues" : $util.dynamodb.toMapValuesJson($ctx.args.input)
}

The animals resolver request template on the Farm type will look similar to cows or chickens but it will use the prefix Animal# and the GS02 index.

{
  "version" : "2017-02-28",
  "operation" : "Query",
  "index": "GS02",
  "query" : {
      "expression": "PK02 = :pk",
      "expressionValues" : {
        ":pk" : $util.dynamodb.toDynamoDBJson("Animal#$ctx.source.id")
      }
  }
}

This will return data for both Cow and Chicken types sorted by the animal name. AppSync can tell which type each record is by looking at the __typename value that was written when it was created.

Before finishing I want to share a couple of tricks that you might find helpful.

Storing the lower case value of a field in SKxx gives case insensitive sorting while still being able to show the value as originally provided.
Using a default value like AAAAAAAAAAA or ZZZZZZZZZZZ in SKxx when a field is null allows you to group entries without a value at the top or bottom of a list that is otherwise sorted.
Adding prefixes to the SKxx value can be used to group sorted results. An example of this would be prefixing the name of active users with A and inactive with I.
Conditionally writing attributes allows you to create sparse indexes when you want to exclude records from the result. An example of this might be using GS01 to access all cows that the farm has ever owned and GS02 to access only cows that are still here.

Many to Many Relationships

Many to Many relationships with DynamoDB and AppSync can be problematic. Where possible you should avoid them. This can be done by:

Denormalizing data. For example: Store the state and country name as part of an address record instead of storing ID's then using resolvers to look up those records.
Perform the join in the application. For example: A CMS might have a many to many relationship between content and content types but it only has a small number of content types. You could load all of the content types through a list operation (using a DynamoDB Query) then match the content to the content type in the application.

If you cannot avoid many to many joins then options include:

Using 1-M and M-1 relationships with an intermediate type (recommended by the Amplify team). This works but it should come with a big warning because you are almost certainly introducing the N+1 problem. On the 1-M side you'll be using an efficient Query operation but on the M-1 side you're probably performing M x GetItem operations.
If you only have a small number of connections between two items and you only access it in one direction then you might be able to use a DynamoDB list attribute with BatchGetItem. Example: I might have an Image type that stores a larger number of images. Each user needs to pick 3 images. In the DynamoDB record for the user I could store the selected images ID's in a list attribute.

Data Migrations

As your GraphQL types change you will need to migrate data. Typically when this happens you need to find every record of a certain type then update it. Two hacks I've used to reduce the frequency of these updates are:

Returning null and forcing the client to deal with null in a sensible way.
Using the resolver response template to transform the DynamoDB data before sending it to the client.

At some point you're going to need to update data inside DynamoDB. To perform migrations I wrote a script that loads the entire table into memory as an array using the DynamoDB Scan operation. It then looped over every record passing it to a migration function based on the __typename attribute (if found). This function returns the DynamoDB operations that need to be performed to upgrade the record or null if there are none.

This is a simple solution but perfect for adding, removing and updating data.

Comparison to Multiple Table Design

So how does this compare to a multiple table solution like Amplify?

There's surprisingly very little difference.

The number and type of DynamoDB operations is almost the same. The only difference is that you should not Scan a single table solution to return all records of one type.
With on-demand capacity there is almost no price difference.
Both are likely to require updating items when making changes to the schema and can use the hacks I suggested.

The big differences are:

The number of tables you need to manage/monitor.
Multiple return types work better with single table.
Adding and removing DyanmoDB GSI's occurs less frequently with the single table design.

Can You Build a Better Single Table Solution?

Yes.

This solution is designed to be generic and contains compromises to achieve that. By looking at your access patterns you can design a solution that is more efficient.

What Next?

I've already started work on version 2 that will include:

Better support for many to many relations
Delta Sync support
Amplify data store support

I'm also working on a tool to generate a single table SAM backend.

If you want to know more then subscribe to my mailing list below or over at Graphboss.

How do I get my API Gateway URL?

Rich — Fri, 07 Apr 2017 00:00:00 +0000

Occasionally you need to know the API Gateway URL for your services inside your Lambda. This happened to me recently when one of my Lambda's needed to provide a callback URL to a third party service that it was using.

It seems that a lot of people are solving this problem by deploying their API's using Serverless, then copying the URL and redeploying again with that URL hard coded into their serverless.yml as an environment variable.

If you're one of those people then STOP. This is both the hard way and it has two very negative side effects.

It's difficult to deploy your application to other stages, and
It introduces a risk that you'll break something by depolying to one stage with the API Gateway URL for the different stage.

So how do I get around this?

Instead of hard coding the API GW URL you can reconstruct it using a combination of Serverless variables and CloudFormation. Here's how I set the API gateway URL so it's passed to my Lambda as the GW_URL environment variable.

provider:
  environment:
    GW_URL:
      Fn::Join:
        - ""
        - - "https://"
          - Ref: "ApiGatewayRestApi"
          - ".execute-api.${self:custom.region}.amazonaws.com/${self:custom.stage}"

Inside my code I can now reference the current API GW URL using.

process.env.GW_URL;

If you're using the serverless-offline plugin (or similar) then you will probably want to combine this with per stage environment variables. You'll need to use per stage environment variables because this approach assumes you're deploying with CloudFormation but running it locally means no CloudFormation to resolve the URL. Instead you'll need to hard code a URL for local development.

The "cheat sheet" version is below but I strongly recommend you read the post as the full version explains it in more depth and covers topics like having separate files for each environment.

provider:
  environment:
    GW_URL: ${self:provider.${self:custom.stage}.GW_URL}

custom:
  stage: "${opt:stage, self:provider.stage}"
  prod:
    GW_URL:
      {
        "Fn::Join":
          [
            "",
            [
              "https://",
              { "Ref": "ApiGatewayRestApi" },
              ".execute-api.${self:custom.region}.amazonaws.com/${self:custom.stage}",
            ],
          ],
      }
  dev:
    GW_URL: "http://localhost:3000/"

Using this technique you can also export your API Gateway URL from your CloudFormation stack so that it can be imported into other stacks. This is one way to solve the service discovery problem.

resources:
  Outputs:
    ApiUrl:
      Description: "The API Gateway URL"
      Value:
        Fn::Join:
          - ""
          - - "https://"
            - Ref: ApiGatewayRestApi
            - ".execute-api.${self:custom.region}.amazonaws.com/${self:custom.stage}"