DEV Community: Chris Armstrong

Visual Regression Testing with Storybook 7

Chris Armstrong — Sat, 13 May 2023 07:38:49 +0000

Storybook 7 has deprecated the storyshots plugin, which was commonly used to set up a visual regression testing workflow with jest-image-snapshot (as an alternative to their paid Chromatic service, which offers storybook hosting and visual regression testing).

However, at the time of writing (13/5/2023), Storybook are yet to update their documentation to say this is deprecated, with it only being mentioned in Github issues, and no alternative workflow documented with their new test-runner yet. It is even broken in v7, with recommendations to use workarounds instead of a mainline fix yet.

Overview

The new test-runner functionality uses playwright and jest under the hood to run in-browser tests against each of your stories. We can hook this new functionality and run a visual snapshot test with jest-image-snapshot.

Each story in your storybook is transformed into a test using babel, and you can add render assertions with a play() function in your story.

We will customise the underlying test-runner configuration to add a post-test hook to create a snapshot with jest-image-snapshot.

Install dependencies

yarn install jest-image-snapshot @storybook/test-runner -D

Creating a test-runner config

Storybook have thankfully provided useful jest configuration hooks we can use to install jest-image-snapshot and run it after each generated story test.

Create a file in your storybook directory at .storybook/test-runner.ts with this configuration:

import { TestRunnerConfig } from '@storybook/test-runner'
import { toMatchImageSnapshot } from 'jest-image-snapshot'

const config: TestRunnerConfig = {
  setup: () => {
    expect.extend({ toMatchImageSnapshot })
  },
  postRender: async (page, context) => {
    // Add a post-render delay in case page is still animating
    await new Promise((resolve) => setTimeout(resolve, 500))
    const screenshot = await page.screenshot()
    expect(screenshot).toMatchImageSnapshot({})
  },
}

export default config

Add a babel configuration

If you don't already have a .babelrc (or equivalent), you will need to create one (e.g. if you use swc already, you will still need to create a babel configuration as unfortunately Storybook test-runner needs it to read your
.stories.tsx files and create jest unit tests out of them).

.babelrc

{
  "presets": ["@babel/preset-typescript", "@babel/preset-env"]
}

Running visual regression tests

The visual regression tests can be setup in your package.json scripts section:

"scripts": {
  ...
  "test-storybook": "yarn run storybook-test"
}

When running the test-runner you need an instance of storybook running in the background:

yarn run storybook &

and then run test-storybook:

yarn run test-storybook

Establish a baseline

When you run your visual regression tests for the first time, storybook-test will create a baseline. This will save image files under __image_snapshot__ in your source code folder, which you should check in.

Updating your snapshots

You will need to update your snapshots when there has been a valid change to the layout of your story or there has been a regression.

When a regression is found, the test-runner will fail and it will give you an image path on your filesystem you can reference to see what the regression was.

Once you have fixed and validated your stories and are satisfied they are working correctly, you can update your
baseline snapshots by running:

yarn run test-storybook -u

and check in the resulting changes to git.

Running visual regression tests in CI/CD

You can use the method above (running storybook) to execute the visual regression tests in a CI/CD pipeline.

If you build your storybook for deployment to a server, you can also execute the tests against that server or against the built artifacts with a local http server.

For example, to run them locally after building the storybook with yarn build:

yarn add wait-on concurrently http-server -D
npx concurrently -k -s first -n "SB,TEST" -c "magenta,blue" \
  "npx http-server storybook-static --port 6006 --silent" \
  "npx wait-on tcp:6006 && yarn test-storybook

API Mismatch: Why bolting SQL onto noSQL is a terrible idea

Chris Armstrong — Thu, 12 Jan 2023 00:20:21 +0000

TLDR; Use abstractions that are designed to hide the complexity of the underlying technology, not those that expose its limitations and do not match its semantic model.

I recently started experimenting with Remix for a new side-project idea using a tutorial with a starter template, that happened to include Prisma (a SQL data access layer, self-described TypeScript ORM) with sqlite as its database.

I've not had that much to do with Prisma, but its API seems self-explanatory. The automatic schema management, and the schema migration feature, seem like an excellent way to manage new tables and indexes you add to your code through the deployment cycle (something many ORMs don't give you).

This got me thinking about DynamoDB (a database I use day to day, and for which I maintain dynaglue, a single-table mapping layer for TypeScript/JavaScript), and made me wonder if a DynamoDB adapter existed. I stumbled across this GitHub issue, and arrogantly shot off this tweet not expecting any replies.

But DynamoDB supports SQL

As one commenter pointed out, DynamoDB supports PartiQL, which helps provide something of a SQL like interface to DynamoDB, which is completely true, and on the face of it, would be beneficial for creating a DynamoDB adapter for ORM layers like Prisma.

What that conceals from the discussion, however, is that the supported subset of PartiQL is only that which maps directly onto DynamoDB design. That subset is no-where near the set of features that are considered standard in a SQL database.

For starters, APIs like groupBy, count and aggregate, update and updateMany have no DynamoDB equivalent (and hence no PartiQL equivalent). They can be simulated by an adapter, but that means Prisma is no longer behaving as an ORM, but is inheriting responsibilities of the database itself.

Much worse, is the fact that PartiQL will supports parts of findUnique, findMany, etc., but unless the developer specifies a primary key field in their WHERE clause, it will fallback to using a scan on the table. A scan can be reasonably fast for a small number of items, but becomes linearly slower on larger tables.

Developers working with relational databases will be well aware of this problem - a query that performs adequately in development (because there are few items in your table) will perform atrociously in production, the fix being simply to add an index. This is the case for DynamoDB as well, but there are limits on how you can index existing documents (you will need a data migration if you need to index a nested value).

If pay-per-request pricing is used with DynamoDB, this scan can also be expensive, as users are charged per items scanned in the partition, not what is returned in the response. (When provisioned capacity is used, the developer is more likely to saturate the available read units, especially if parallel scanning is deployed). A relational database will just degrade in overall performance until it is scaled upwards.

Do you have a plan?

Relational databases are smart enough to work out what indexes they have available to them and "plan" how to execute a query based on what is available. This means they may be able to satisfy the entire WHERE clause using an index, fall back to scanning and index and then filtering the results, combining and filtering the results of looking up or scanning multiple indexes, or scanning and filtering the entire table.

The decisions a query planner makes about what indexes to use or not, comes down to both its knowledge of those indexes, and the statistics collected on the use of those indexes. This allows for a relatively sophisticated query execution based on the database engine's initimate knowledge of the underlying data storage and its ongoing usage.

The statistics a SQL engine can store includes how long it takes to access an index based on certain query parameters, how much data is populated in that index, etc., which are all used to inform the query planner's decision making process.

On the other hand, a DynamoDB database provides none of this to PartiQL (its not even available to client applications). At best, it can identify indexed fields in the WHERE clause, and select indexes that may help to speed up the query by reducing the set of results to scan.

However it won't be able to make any sophisticated decisions on how to use those indexes. It does not have any information about how sparsely those indexes are populated, relative lookup times, etc. that a relational engine would normally use to speed up execution. In most cases, its going to fall back to a full database scan.

Why a DynamoDB scan is not the same as a relational SQL scan

In my mind, this isn't even the worst part, which boils down to a fundamental difference is how data is stored and retrieved in DynamoDB vs relational databases.

In relational engines, the query is executed very close to the where the data is stored (typically on the same node). This means that a scan can be relatively fast, even over a large number of items, because the retrieval and filtering is performed on the database node before a response is streamed back to the application.

On the other hand, DynamoDB will perform filtering on the node, but it does it in fixed size chunks. The application must wait an entire round-trip over the network to request a scan on the next part of the index. This problem is made much worse when most of the index's elements are filtered out.

The relational database only has to wait between the memory and the storage drives (or if splitting over a cluster, for batches of results to come back from each node). Each round-trip from the application and the database can be optimised to return as much as possible, which is not possible with DynamoDB.

(The limits on scanning can be ameleorated using a parallel scan, but this is best done as a deliberate implementation detail where the potential costs and resource usage are considered beforehand, not as an off-hand consequence of an untested query).

Abstractions and depth of understanding

The value in a layer like Prisma is that it abstracts away the hairier details of a database engine from the developer, leaving the indexing and query writing to the underlying adapter. Developers can focus on business logic instead of technical details.

When the database layer underlying the Prisma API cannot satisfy the majority of the API, or runs with unexpected performance and cost issues when the supported APIs are found, the value underlying the abstraction is lost. The developer now has to understand the limits of DynamoDB and be careful what they query to make sure it is optimally using indexes.

This is something where a relational database is more forgiving (just add an index when performance degrades). For developers that does understand the limits of DynamoDB, they will then be needing to migrate data to support new indexes and "drop-down" to the DynamoDB interface itself when they cannot perform what they need in the database adapter.

(I did mention before that it would theoretically be possible to build the missing functionality into the Prisma adapter itself. There may be value in building a SQL like interface over DynamoDB, but you're sacrificing one of the reasons for using it in the first place - consistency of performance under load - while absorbing the cost of maintaining what would normally be a battle-tested component of the database itself)

Use the platform, not the abstraction

Does this mean you should just use a relational database?

This is where I think technology selection is important. NoSQL databases like DynamoDB have the advantage of consistent performance and automatic scaling under increasing load, which is not something that can normally be said of relational databases.

The trade-off, of course, is learning how to use the technology, ensuring that your access patterns are well suited for NoSQL databases, and designing your data storage to optimise queries with those access patterns. You need to understand the limits of the platform and use other systems to fill those requirements not fulfilled by DynamoDB.

That said, if your team's experience is with relational databases, or your application is not suited for NoSQL workloads, SQL databases offer an excellent solution backed by decades of refinement, flexible storage options and well understood limits and scaling requirements. You're in safe hands backed by broad institutional knowledge of the technology (you'll never have a problem finding a developer who has worked with SQL).

Throw Prisma in the mix, and you have a well-worn abstraction (with the additional benefits of schema management that it brings) that will serve your needs well and let your developers focus on what matters.

Package your NodeJS Lambda functions individually with esbuild for faster cold-start times

Chris Armstrong — Wed, 05 Jan 2022 01:08:31 +0000

Learn about packaging your NodeJS-based AWS Lambda functions with esbuild for improved cold start times and reduced deployment package size

This article was originally posted on my development blog, where you can find other articles about TypeScript, AWS, Lambda and open source.

Introduction

AWS SAM is a great way to package and deploy your AWS Lambda functions, but as your project grows, it's easy to become frustrated with increasing deployment times from an ever-growing deployment package containing all your functions and NodeJS dependencies.

The serverless framework supports an excellent workflow with the serverless-webpack plugin to package your functions individually with webpack. For AWS SAM, it's probably worth checking out the aws-sam-webpack-plugin, but I've found it far simpler to configure esbuild to perform the same task.

Why package my code using a bundler?

Improved cold start time - smaller deployment packages generally means less time copying your deployment package to the AWS Lambda container each time it has to be started
TypeScript support - NodeJS doesn't support TypeScript natively, so converting it to JavaScript can be done as part of a bundilng step
Tree-shaking and reduced deployment package size - a bundler will only include the node dependencies your function imports, and if enabled correctly for tree-shaking, your overall bundle size will be reduced to just the blocks of code that are needed (which is great for multi-functional libraries like lodash which are rarely used as a whole).

Disadvantages

Using a bundler does come with some downsides, including:

Increased complexity - bundling your code means more steps before running your code locally using aws sam local and more deployment time steps
Inceased packaging time - as your functions are packaged individually, a unique deployment package is generated for each one, instead of uploading a single deployment package with all of your functions. However, this may still be acceptable as each function's deployment package will be considerably smaller after bundling
Poor stack trace support - getting decent stack trace support with bundled code can be very difficult, especially for exceptions. In many cases, you'll still be debugging by console.log() or guessing based on the function names in your stack trace.
Module support - some node modules are not compatible with bundling, either because they assume the existence of the package.json and node_modules folder (looking for filesystem artefacts in their code), or they are trying to perform auto-instrumentation for telemetry (which bundling interferes with).

esbuild vs webpack

I've used webpack before for bundling, and while it is a mature and flexible bundler, you may run into problems with its memory usage and execution when you use it on very large projects with dozens of AWS Lambda functions. This is because it is written in JavaScript and must execute the same bundling process for each Lambda function.

esbuild is a newer bundler built in Go, which is gaining considerable popularity for its speedy execution time and simple configuration, as well as removing the need for Babel with its native TypeScript support. However, it should be noted it has less features and is less mature than webpack (and probably always will because it is deliberately targeting a reduced feature set).

I've chosen esbuild because I've found it mature enough to use in production for large, established AWS Lambda based applications, and it reduced bundling times from over 5 minutes to less than 30 seconds.

Using esbuild with AWS SAM

Configuring your project

In this setup, we will bypass sam build and set up an esbuild bundling step that must be run before every sam package/sam deploy.

Install esbuild

First get esbuild installed with npm or yarn:

npm i esbuild -D

# OR

yarn add esbuild -D

Source code layout

Next structure your function entry points to be hosted in individual directories - this helps with ensuring that bundled output for each function gets its own directory, and hence can be deployed individually by SAM.

e.g.

src/say_hello/index.ts
src/send_email/index.ts

(I've assumed TypeScript here, but you can use JavaScript by simply changing the file extension. Remember that you should have typescript installed in your project and a tsconfig.json set up for your Node version if you're transpiling TS).

If you have any other shared code, it should be put into another directory e.g /shared or /util and simply imported.

Setup esbuild configuration

Although esbuild can be run from the command line, a quick JavaScript configuration file is easier to maintain and read with the number of options we will use.

const fs = require('fs');
const path = require('path');
const esbuild = require('esbuild');

const functionsDir = `src`;
const outDir = `dist`;
const entryPoints = fs
  .readdirSync(path.join(__dirname, functionsDir))
  .map(entry => `${functionsDir}/${entry}/index.ts`);

esbuild
  .build({
    entryPoints,
    bundle: true,
    outdir: path.join(__dirname, outDir),
    outbase: functionsDir, 
    platform: 'node',
    sourcemap: 'inline',
  });

This assumes your Lambda function entry points are located in ./src/<function_name>/index.ts and will be built to ./dist/<function_name>/index.js.

If you would prefer to do the above using the command line, the following esbuild line should be equivalent:

npx esbuild \
  --bundle src/*/index.ts \
  --outdir=dist \
  --outbase=src/ \
  --sourcemap=inline \
  --platform=node

Setting up your AWS SAM template

Each function in your AWS SAM template will need to be updated to use the new package layout:

Resources:
  HelloFunction:
    CodeUri: dist/say_hello
    # assumes your function is at `src/say_hello/index.ts` or 
    # `src/say_hello/index.js` with an exported entry point 
    # function called `handler`
    Handler: index.handler
    ...
  SendFunction:
    CodeUri: dist/send_email
    Handler: index.handler
    ...

Building and deploying your function

You can now build and deploy your function by running esbuild before each sam package and sam deploy step i.e.

node esbuild.js 
sam package
sam deploy

Running locally

Your bundled code can be executed locally, but you need to run node esbuild.js before you call sam local invoke.

If you want to re-run esbuild in watch mode, you can modify your esbuild.js accordingly:

esbuild.
  build({
    ...
    watch: process.argv.includes('--watch'),
  });

and run node esbuild.js --watch in the background (or add --watch to the command line variant given in Setup esbuild configuration).

SQS Queues as an EventBridge Rule Target (with CloudFormation)

Chris Armstrong — Thu, 02 Dec 2021 06:40:28 +0000

AWS offers a dizzying array of targets for EventBridge Rules, but even if you set them up through the console, the documentation for each target type can be a bit light.

SQS, being an early AWS service, can come with some extra quirks that make integrations a bit trickier, especially when it comes authorization or configuration.

In this article, I show you how to set up SQS queues and SQS FIFO queues as a target of an Event Bridge Rule using CloudFormation syntax.

NOTE: I've assumed you're familiar with EventBridge events and SQS, and just want to get up and running.

Event Bridge to SQS Queue - the simple case

There is nothing special in the EventBridge rule - just the target ARN of the Queue:

  MyQueueTarget:
    Type: AWS::Events::Rule
    Properties:
      # Configure the event pattern to filter your events (https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html)
      EventPattern:
        source: ["com.mycompany.events"]
        detailType: ["CustomerPurchase"]
      Targets:
        - Arn: !GetAtt MyQueue.Arn
          Id: QueueTarget # set as needed

You don't need a role ARN because the permissions are handled using queue policies

Queue and Queue Policy

Define your queue - it requires no special properties:

  MyQueue:
    Type: AWS::SQS::Queue

and define a queue policy, which will permit EventBridge to write events to your queue:

  MyQueueEventBridgePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          -
            Effect: Allow
            Principal: { Service: events.amazonaws.com }
            Action: SQS:SendMessage
            Resource: !GetAtt MyQueue.Arn
      Queues:
        - !Ref MyQueue

You can further restrict the policy with a condition key limiting access to the event rule that invoked the policy (you shouldn't consider this optional, otherwise you're granting access to any EventBridge rule):

         Statement:
           -
            ...
            Condition:
              ArnEquals: { "aws:SourceArn": !GetAtt MyQueueTarget.Arn }

Event Bridge to FIFO Queue - the complex case

Similar to the normal SQS queue case, you define your event bridge rule:

  MyFifoQueueTarget:
    Type: AWS::Events::Rule
    Properties:
      # Configure the event pattern to filter your events (https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns.html)
      EventPattern:
        source: ["com.mycompany.events"]
        detailType: ["CustomerPurchase"]
      Targets:
        - Arn: !GetAtt MyFifoQueue.Arn
          Id: QueueTarget # set as needed
          SqsParameters:
            MessageGroupId: myMessageGroupId

Similar to SQS queues, you don't need a Role ARN, but you do need to specify a MessageGroupId. This value is a fixed string - there is currently no support to make this dependent on a value in the incoming event, which means all your messages will be in the same message group.

Queue and Queue Policy

Your queue is similar, but will need to be a FIFO queue, and have content-based deduplication turned on (this seems to mandatory to avoid silent failures):

  MyFifoQueue:
    Type: AWS::SQS::Queue
    Properties:
      FifoQueue: true
      ContentBasedDeduplication: true

The queue policy is the same (see the previous section for an example).

What about encryption?

SQS supports two types of encryption at rest:

SSE-SQS - SQS manages the encryption for you. This currently isn't available with CloudFormation, so I haven't tested its use with EventBridge
SSE-SQS - KMS is used to perform encryption, either with a AWS-managed key or a Customer-Managed Key (CMK)

Transport encryption still relies on TLS, and IAM (via queue policies) is used to perform authorization and control access to the queue.

Using the AWS-managed key with SQS (by specifying KMSMasterKeyId: alias/aws/sqs) doesn't appear to work, and this is probably because specific access hasn't been granted to EventBridge to access the key, and probably can't be made to work because AWS managed keys do not let you edit their IAM key policy.

A customer managed key may be able to be made to work - see this FAQ for an example.

Hopefully once SSE-SQS is available in CloudFormation, it should be simple matter of enabling it so encryption can be enabled in SQS when used as an EventBridge target.

Processing messages

The records that arrive in your SQS queue will have a body of the whole EventBridge event by default (unless you configure your EventBridge rule to transform, or select part of the event, or pass a different JSON entirely using the InputTransformer/InputPath/Input properties - see this article for details on using InputTransformer).

If you process your "EventBridge over SQS" events with AWS Lambda, your input event will look something like:

{
    "Records": [
        {
            "messageId": "9c3aeeeef-6eee-4a18-9abc-89753969a233",
            "receiptHandle": "AQEBUjLMNIEkASgzMS9sSrfQdkZZa7tfH0sLH/KEXpr3P3c+Aen0WQ3x3gZoKXycvhkjU3F6vA+VwRky3JOCRxUN5o1d97y52Iqw0ID/HUflIXDoJWEUCRDUIRCONTEtbYW39EE5Fxxz+rc4YyBw5v8dlxCDtXdgdyEucUVIdQt4K+94YJdz3GTpQF2ASg0YdhRQvWQb4to+wfJLVW/C0f/cgY43zFTNHrRCSuRKBMNEUONSKntc0ubh7QHyBjNoIUC+2QXGq2ECdkBlBJ9BVDQZLXuObgjcoL1hi2XiLebTKCRT0Jo0rMEfs17giLHIFwy1ZrKeOqim",
            "body": "{\"version\":\"0\",\"id\":\"5b2fcd39-5fd6-b4ef-51a6-7d5624ced5da\",\"detail-type\":\"CustomerPurchase\",\"source\":\"com.mycompany.events\",\"account\":\"3829362938291\",\"time\":\"2021-12-02T06:01:09Z\",\"region\":\"us-east-1\",\"resources\":[],\"detail\":{\"type\":\"order\",\"orderNumber\":\"2928\"}}",
            "eventSource": "aws:sqs",
            "eventSourceARN": "arn:aws:sqs:us-east-1:3829362938291:my-queue.fifo",
            "awsRegion": "us-east-1"
        }
    ]
}

If you don't care about the EventBridge envelope, and would prefer to receive the raw detail component of the message, the simplest input transformation is to set InputPath to $.detail.

  MyFifoQueueTarget:
    Type: AWS::Events::Rule
    Properties:
      ....
      Targets:
        - 
          Id: QueueTarget # set as needed
          ...
          InputPath: $.detail # just send the event detail

This is most useful when porting existing integrations (e.g. SNS -> SQS with raw message delivery), when you're using your own events, and/or when your event rule selects the Event Source and Event Detail Type such that you know the schema of the detail component without needing the EventBridge metadata to distinguish it.

Testing node-fetch with jest in TypeScript

Chris Armstrong — Mon, 30 Aug 2021 23:16:21 +0000

(This was originally posted on my development blog at chrisarmstrong.dev, so check it out for more JavaScript and AWS related content!)

Because I haven't seen it anywhere else, and because it was a bit tricky to set up, here is an example of testing some code that uses the node-fetch library with TypeScript.

First install fetch-mock-jest, fetch-mock and its types package @types/fetch-mock.

Then, set up your test like follows:

import type { FetchMockStatic } from 'fetch-mock';
import fetch from 'node-fetch';

// We need this import to get the extra jest assertions

import 'fetch-mock-jest';

// Mock 'node-fetch' with 'fetch-mock-jest'. Note that using
// require here is important, because jest automatically 
// hoists `jest.mock()` calls to the top of the file (before 
// imports), so if we were to refer to an imported module, we 
// would get a `ReferenceError`

jest.mock(
  'node-fetch', 
  () => require('fetch-mock-jest').sandbox(),
);

// Cast node-fetch as fetchMock so we can access the 
// `.mock*()` methods

const fetchMock = (fetch as unknown) as FetchMockStatic;

describe('code that uses fetch', () => {
  beforeEach(() => fetchMock.reset());

  it('should fetch a simple URL correctly', async () => {
    fetchMock.get('https://example.com', { value: 1234 });
    await fetch('https://example.com');
    expect(fetchMock).toHaveFetched({ 
      url: 'https://example.com'
    });
  });

  it('should fetch with complex assertions', async () => {
    fetchMock.post(
      'https://example.com/submit', 
      { status: 'ok' },
    );

    const result = await fetch(
      'https://example.com/submit',
      {
        method: 'post',
        body: JSON.stringify({
          type: 'create',
          details: {
            username: 'chris',
            passwordToken: 'abcde12345',
          },
        }),
        headers: { 'content-type', 'application/json' },
      },
    );

    // Check we called the right URL, method and 
    // part of the body
    expect(fetchMock).toHaveFetched(
      'https://example.com/submit',
      {
        matchPartialBody: true,
        method: 'post',
        body: { type: 'create' },
      }
    );
  });
});

Using ReScript to build applications on AWS Lambda

Chris Armstrong — Thu, 26 Aug 2021 12:06:13 +0000

(This was originally posted at my development blog on chrisarmstrong.dev. Check it out for more AWS and DynamoDB related articles!)

Rescript provides a type-safe, compile-to-JS language that delivers blazingly fast compile times with efficient and readable JavaScript output without sacrificing usability.

It's most often used to build front-end applications with React but can also be used to build type-safe applications with NodeJS.

In this tutorial I'll show you how to get up and running quickly with an AWS Lambda application in ReScript. I assume you have some knowledge of ReScript and building for AWS, but in any case I'll explain what is happening as I go through.

The full code accompanying this article can be found on GitHub

Prerequisites

To begin, you'll need to have installed:

NodeJS: this can be installed directly or using a tool like nvm to manage multiple NodeJS installations. I've assumed version 14.x here which is the latest stable version
AWS SAM CLI: Servlerless Application Model (SAM) is one of the various AWS native ways you can develop serverless applications.

You'll also need to have an AWS account and have configured SAM with with your AWS credentials.

Set up project

In a new directory, run npm init to generate a package.json:

mkdir rescript-lambda-example
cd rescript-lambda-example
npm init -y

Next, install some of the modules we will need:
```
npm i rescript esbuild npm-run-all
```
ReScript is installed as an npm module. In addition, we'll be using esbuild to bundle our code and npm-run-all to coordinate our shell scripts.

API Handler

AWS Lambda API handers for NodeJS consist of a JavaScript file that exports a function of the form:


export const handler = async (event, context) => {
  // some handler code
}

where the event is structured based on the Lambda trigger, and the context provides context information
about the Lambda runtime environment (such as the request ID and authentication information for API Gateway
contexts).

Our Lambda will be used with API Gateway, which has events structured like this:

{
  "body": "{\"name": \"Chris\"}",
  "isBase64Encoded": false,
  "headers": {
    "content-type": "application/json",
    "accept": "*/*",
  },
}

We will build a simple hello-world handler that accepts a JSON-encoded body containing an object with a single name field, and echo it back in the response to the HTTP client.

The Code

First, we define the structure of our incoming request body and outgoing response body and serialisation functions for them.

src/NameMessage.res

/**
 * The type of the incoming body
 */
type nameMessage = {name: option<string>}

/**
 * The type of the response body
 */
type nameResponse = {message: string}

/**
 * Decode a JSON-encoded string using JSON.parse()
 * and assume it is the nameMessage type
 */
@scope("JSON")
@val 
external parseNameMessage: string => nameMessage = "parse"

/*
 * Stringify the name response message
 */
@scope("JSON") 
@val 
external stringifyResponse: nameResponse => string = "stringify"

These serialisers are just JSON.parse (mapped as parseNameMessage) and JSON.stringify (as stringifyResponse). These aren't particularly type safe, and as you'll see, we can easily cause runtime errors as they effectively cast whatever input the user provides into the nameMessage structure.

We'll see how we can build type-safe encoders/decoders in a future post, but for now, this wrapping the JSON API directly us get started quickly with our example.

Next, we'll define the structure of the API Gateway event and the expected response from the Lambda event handler:

src/ApiGateway.res

/*
 * An API Gateway Lambda Event.
 *
 * (with a simplified interface with just what
 * we need for this example)
 */
type awsAPIGatewayEvent = {
  body: option<string>,
  isBase64Encoded: bool
}

/**
  * An APIGateway response type. In this we
  * seralise the response body and specify the
  * status code (with any headers, if necessary)
  */
type awsAPIGatewayResponse = {
  body: string,
  headers: Js.Dict.t<string>,
  statusCode: int,
}

These are based on the API Gateway Event structure we showed before.

Next, we'll define some wrappers around NodeJS Buffer.from() and Buffer.prototype.toString() (which we'll need to decode base64-encoded API Gateway bodies).

Lastly, we use the above two files and define our handler:

src/Hello.res

open Belt.Option
open ApiGateway
open NameMessage

let handler = (event: awsAPIGatewayEvent): Js.Promise.t<awsAPIGatewayResponse> => {
    Js.log2("body", isNone(event.body))
  // Convert body to a UTF-8 string if base64-encoded
  let body =
    event.body
    ->map(body =>
      event.isBase64Encoded
        ? body->Buffer.from(~encoding=#base64)->Buffer.toString(~encoding=#utf8)
        : body
    )
    ->flatMap(body => body != "" ? Some(body) : None)

  // Parse the event body string using JSON.parse
  let message = map(body, parseNameMessage)

  // Extract the name field
  let name = flatMap(message, message => message.name)

  // Construct the response body
  let responseBody = {
      message: `Hello, ${getWithDefault(name, "there")}!`,
  }

  let response = {
    statusCode: 200,
    headers: Js.Dict.fromArray([("content-type", "application/json")]),
    body: stringifyResponse(responseBody),
  }
  Js.Promise.resolve(response)
}

Our handler starts by decoding the body (if it happens to be base64-encoded) and checking if it is an empty string (making it None if it is to short-circuit the remaining code).

It then uses parseNameMessage to parse the JSON-encoded body string, and then extracts the name property.

(Note that we use map and flatMap (from Belt.Option) throughout, as the event.body value is optional.)

Lastly, we construct the response body object, and then stringify it as part of the API Gateway response (which
also includes the Content-Type: application/json header and 200 status code).

Note that if the name value is not specified (i.e. it is the special value None), we use there as the default mapping. This is to account for when a body is not provided by the caller.

Configuration

We need to configure two aspects of this project - the Rescript build and the serverless application deployment.

Building your code with Rescript

The Rescript build is mostly boilerplate, but we'll cover it here to identify the main customisation points.

The configuration I demonstrate uses a bundler, which most consider unnecessary for server-side nodejs code, but which is a good idea in AWS Lambda environments as it both drastically reduces your Lambda start up time and the time it takes to upload your code for deployment. It can also help improve the performance of your Lambda overall as there is far fewer filesystem reads from loading the required source files.

The bundler I've chosen is esbuild, which is a great complement to ReScript as it is also incredibly fast.

bsconfig.json

Create a new file called bsconfig.json in the main project directory and fill it as follows:

{
  "name": "rescript-lambda-example",
  "sources": { "dir": "src" },
  "package-specs": {
    "module": "es6-global"
  }
}

Breaking it down:

name - this the name of the project, which is unused here because we a compiling an application, but in the case of libraries, it sets the namespace
sources - set the source directory (this can also be an array when there are multiple source directories)
package-specs.module - set the output to ES6 modules using import/export style, instead of the default ES5/commonjs output. This ensures that the bundler (esbuild) can tree-shake out unused functions from our imports.

package.json

In the package.json, add the following new scripts:

  "name": "rescript-lambda-example",
  "scripts": {
    "build": "run-s build:rescript build:bundle",
    "build:rescript": "rescript build",
    "build:bundle": "esbuild --outdir=build --target=node14 --platform=node lib/js/src/Hello.js",

    "develop": "run-p develop:rescript develop:bundle",
    "develop:rescript": "npm run build:rescript -- -w",
    "develop:bundle": "npm run build:bundle -- --watch"
}

We've added two main scripts: build, which does a full build and bundle (for CI/CD and deployment), and develop, which uses a watch mode to continuously rebuild and rebundle. npm-run-all has been used to delegate to two other npm scripts and run them sequentially (run-s) or in parallel (run-p) as needed.

Both scripts run by first using rescript to compile the code (which is output to lib/js) and then bundle using esbuild which outputs to the build directory.

The ReScript documentation has more information on configuring bsconfig.json and the rescript --help and esbuild --help will tell you all you need to understand their respective options.

Setting up a deployment template

SAM uses an extension of AWS CloudFormation Templates for its configuration.

Create a new file called template.yaml in the root project directory with the following declarations:

Transform: AWS::Serverless-2016-10-31

Resources:
  IndexHandler:
    Type: AWS::Serverless::Function
    Properties:
      Handler: Hello.handler
      CodeUri: build
      Runtime: nodejs14.x
      Events:
        ApiEvent:
          Type: HttpApi
          Properties:
            Method: POST
            Path: /hello
Outputs:
  ApiGatewayURL:
    Value: !GetAtt ServerlessHttpApi.ApiEndpoint
    Export:
      Name: !Sub "${AWS::StackName}-url"

This short template:

Generates a Lambda function from our build/Hello.js file (specifying the entry point as the handler function)
Adds a HttpApi event trigger to the function for path /hello and HTTP method POST
Implicitly generates an AWS API Gateway HTTP API with default deployment and stage (based on the HttpApi event - the logical ID of the API is ServerlessHttpApi)

Test Handler Locally

We can test our handler locally by using the sam local command (you will need to have Docker installed for this to work):

# Build the application
> npm run build
# Start a local HTTP server
> sam local start-api

SAM should spin up a HTTP server on localhost:3000. We can run a quick testing using curl in another terminal session:

# Send a payload to curl (the POST HTTP method is implied)
> curl http://localhost:3000/hello --data '{"name": "Chris"}'
{"message":"Hello, Chris!"}%

Note that the first time you run this may take a while, as SAM will need to download and build a Docker image to run your Lambda within.

We can also see what happens when we don't provide a request body:

> curl http://localhost:3000/hello -XPOST
{"message":"Hello, there!"}%

We can also show how the JSON.parse wrapper is just as un-type-safe as JavaScript:

# Send a payload to curl (the POST HTTP method is implied)
> curl http://localhost:3000/hello --data '{"name": {}}'
{"message":"Hello, [object Object]!"}%

Deploy to the environment

The first step is packaging, which uploads your code and transforms the template to refer to its S3 location.

sam package \
  --resolve-s3 \
  --output-template-file template.deploy.yaml

Once that is done, we can run the deployment command. This will create a new CloudFormation stack containing your Lambda and API Gateway.

sam deploy \
  --template-file template.deploy.yaml \
  --resolve-s3 \
  --stack-name rescript-example \
  --capabilities CAPABILITY_IAM

The output should contain something like the following, which identifies the URL where your API is deployed:

CloudFormation outputs from deployed stack
---------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs                                                                                                                                                 
---------------------------------------------------------------------------------------------------------------------------------------------------------
Key                 ApiGatewayURL                                                                                                                       
Description         -                                                                                                                                   
Value               https://2uhonte28e.execute-api.us-east-1.amazonaws.com                                                                         
--------------------------------------------------------------------------------------------------------------------------------------------------------------

We can then use curl again to test our deployed API to verify it is working correctly:

> curl https://2uhonte28e.execute-api.us-east-1.amazonaws.com/hello -XPOST --data '{"name": "World"}'
{"message":"Hello, World!"}%

Next Steps

In a future post, we'll look at:

Improve JSON handling
Add multiple endpoints
Make AWS API calls

Retry, Timeout and Cancel with fetch()

Chris Armstrong — Sun, 22 Aug 2021 11:13:34 +0000

(This was originally posted at my development blog on chrisarmstrong.dev. Check it out for more AWS and DynamoDB related articles!)

Although none of these are built-in, it's easy to add retries, timeouts and cancellations to your fetch() calls without needing to use a full-featured third-party library.

Retries

By wrapping your fetch handler in a recursive function that returns a promise, you can easily get retry behaviour:


const fetchWithRetries = async (
url, options, retryCount = 0) => {
  // split out the maxRetries option from the remaining options
  // (with a default of 3 retries)
  const { maxRetries = 3, ...remainingOptions } = options;
  try {
    return await fetch(url, remainingOptions);
  } catch (error) {
    // if the retryCount has not been exceeded, call again with retryCount + 1
    if (retryCount < maxRetries) {
      return fetchWithRetries(url, options, retryCount + 1)
    }
    // max retries exceeded, just throw error
    throw error;
  }
}

You could even customise it further with additional flags to control when to retry, or to throw a MaxRetriesError when the maxRetries count is hit.

Timeout

This one is fairly easy - we use setTimeout to reject the promise when there is a timeout error.


// Create a promise that rejects after `timeout` milliseconds
const throwOnTimeout = (timeout) => 
  new Promise((_, reject) => 
    setTimeout(() => 
     reject(new Error("Timeout")),
     timeout
    ),
  );

const fetchWithTimeout = (url, options = {}) => {
  const { timeout, ...remainingOptions } = options;
  // if the timeout option is specified, race the fetch call
  if (timeout) {
    return Promise.race(fetch(url, remainingOptions), throwOnTimeout(timeout));
  }
  return fetch(url, remainingOptions);
}

We use Promise.race to run both the fetch() call and the setTimeout() call at the same time - whichever resolves or rejects first will resolve or reject the Promise (respectively), with the other result being ignored.

Cancel

This one is documented on MDN, but for completeness here it is below.

const fetchWithCancel = (url, options = {}) => {
  const controller = new AbortController();
  const call = fetch(url, { ...options, signal: controller.signal });
  const cancel = () => controller.abort();
  return [call, cancel ];
};

In the above, we return a tuple with the returned promise and a cancel function that allows the user to cancel the request if needed.

An example showing its usage is below:

// We don't await this call, just capture the promise
const [promise, cancel] = fetchWithCancel('https://cataas.com/cat?json=true');

// await the promise to get the response
const response = await promise;

// ...
// cancel the request (e.g. if we have rendered something else)
cancel();

Building successful transactional applications on DynamoDB

Chris Armstrong — Wed, 18 Aug 2021 00:47:40 +0000

(This was originally posted at my development blog on chrisarmstrong.dev. Check it out for more AWS and DynamoDB related articles!)

Building a successful DynamoDB solution

Learning DynamoDB is hard work! But I hear from many developers who have dived in and tried it out because they hear about its performance and scalability and its simple API-model, but back away when it stops working because they've tried to use it like a SQL database, found it too difficult to evolve to a changing understanding of requirements, and simply not invested the time to learn how to use it well.

There are now numerous courses and tutorials covering DynamoDB, which has grown off the back of increasing use of serverless architectures (where DynamoDB is a great fit) and the interesting engineering challenge that comes from using single-table design (popularised by Rick Houlihan's AWS re:Invent talks).

That said, I've gathered here my own recommendations for learning about DynamoDB and strategies I've found useful for using it efficiently and practically.

Understand its capabilities and its limits

Spending the time to understand DynamoDB's capabilities and limits will serve you well when you start modelling a new or existing application's data structures. This will make it easier to decide how to store certain types of data to optimise its access.

the difference between partition and sort keys and how they work together to index an item
what are secondary indexes and how they are used to enable predictable fast access to items without their primary key
the different ways to read data, such as:
knowing about the existence of change streams and transactions and what use cases they solve
having some idea of operational capabilities, such as backups, point-in-time-recovery, encryption, VPC endpoints, etc. which you may never need but are good to know about when your boss inevitably asks you about them.

AWS provides a list of excellent resources for learning DynamoDB in their documentation, but curiously neglects this excellent book on DynamoDB modelling which covers everything above in from beginners to professional-level.

Learning the basics of single-table modelling

Single-table modelling, even if you still decide to use multiple tables, is necessary to know in order to make the most of DynamoDB and its performance capabilities. It's necessary to be able to efficiently implement some one-to-many and many-to-many access patterns, and structuring your data so that your tables can store disparate data types will make it much easier to evolve your application with new data types as the need arises.

Alex Debrie gives a good overview of the what and when of single-table modelling, Rick Houlihan's various AWS:ReInvent talks on advanced modelling are a good way to get a sense of the possibilities while scaring yourself silly then replaying them and pausing every so often, but the best way is probably Alex Debrie's book or one of a number of articles that are available online (such as this one by Forrest Brazeal or this deep-dive video series from AWS).

You probably won't use every single-table modelling technique from the get-go - the one's I've found most useful to be across are:

Understanding how multiple collections of data can be stored in one table
Denormalisation techniques for one-to-many relationships
Storing data and indexes separately by duplicating indexed fields into separate index properties instead of trying to share them (a management layer like dynaglue does this automatically; see the next section)
Strategies for many-to-many relationships, including how to keep join-collections up to date

Use an abstraction layer for table-access and queries

Object-relationship modelling (ORM) frameworks have a bad reputation for increasing complexity without improving the usability of SQL databases from application code, and rightly so, but at the same time they've "poisoned the well" (so to speak) for the advocacy of tools that abstract away from the database layer.

Instead, what has emerged is either techniques for building directly against the application layer, or the adaption to database technology to accommodate the desires of application developers, such as no-SQL databases, the ability to store unstructured JSON data in database columns and query it with SQL, JSON-schema support in the database layer, etc.

With DynamoDB, most new features have been enrichments to existing behaviour, not new ways of querying data that make it easier to query your data model (beyond things like PartiQL which give a familiar SQL interface to DynamoDB but also hide non-performant queries instead of making them obvious to the developer).

I've found using a lighter abstraction layer very helpful with DynamoDB, especially when dealing with single-table designs, as its made it much quicker and less error-prone to create my queries and ensuring that my indexes are populated correctly.

It's also helped remove a lot of the complexity and issues coding directly against the DynamoDB API, but without hiding any performance problems (which is easy to do with PartiQL).

As a result, I developed dynaglue for this purpose, and highly recommend the use of something like it when developing with DynamoDB (alternatives I've seen include DynamoDB Toolbox, TypeORM and DynamoDB OneTable).

Use Schema Validation on your collections to maintain data integrity

You should know what you are writing and reading back from your tables, and I would argue that using a type-aware language is not enough, especially if you are just casting between a JSON structure and an internal type in TypeScript, which gives you no protection against discrepencies in data storage from different pieces of code accessing the same data, or if you need to perform transformations before you read/write data.

Just like you would declare a schema for your SQL tables using DDL, maintaining a JSON-Schema and validating against it before writing to the database, helps to maintain data integrity. It makes sure that you do not accidentally introduce new properties without validating their type first or changing it's definitions inadvertently between code versions. The structure of your data becomes explicit and visible during code changes.

If working in JavaScript, the ajv library provides fast and compliant JSON-schema validation. You can declare your schemas in code and run them just before writing to your DynamoDB table.

(On an additional note, although it would seem convenient, I would caution against re-using schema declarations you may have created for your API in your database schema validation (or vice-versa): sharing schemas for two different purposes makes it hard to evolve them separately, and introduces the risk you would change one or the other in incompatible ways. Always maintain separate separate schemas for your data storage and API layers - future you will be thankful!)

Use appropriate key generation for your primary keys

One of the ways you help maintain a performant DynamoDB table is by ensuring that your partition keys are as unique as possible, so that DynamoDB can distribute your documents across multiple partitions as your data grows. This is obviously hard with timeseries data (the DynamoDB manual has specific advice for handling time-series access patterns), but it is relatively straightforward for typical transaction records.

The ideal document identifier can be generated in a distributed fashion without hash collisions and is reasonably random that it is hard to guess and will shard the DynamoDB at high data storage levels. Even if you are not planning to build tables that run to hundreds of gigabytes, you should still ensure that your keys are random enough to avoid hash collisions.

Another desirable property of identifiers is being naturally sorted based on creation order, which can be achieved with identifiers that are auto-incrementing or encode a timestamp into them.

In SQL databases, an incrementing counter is often used, but this is hard with DynamoDB, especially with distributed clients, because DynamoDB does not have native functionality to produce these values, and generating them in a distributed fashion is difficult. SQL databases can do this efficiently because there is typically one server and these types of operations are done internally (as with all other query operations, such as access-planning, which is why SQL has more trouble scaling in a predictable manner, as hetrogenous and workloads are handled on the same compute layer).

Using timestamps is a poor alternative as well, as even with millisecond resolution, the chance of hash collisions is quite high, especially if you are writing records at a rapid rate, and the records will be grouped close together. I've seen this pattern before, and the result is ugly - it's very easy to override existing records without realising and generating bizarre long-lived behaviour in your system caused by the hash collision.

UUIDs are a step up, but can be a bit cumbersome to use in their typical string representation as they are quite long being encoded in hex. They also don't have any natural sort ordering, but they are a reasonable start to generating unique identifiers in a safe way.

MongoDB uses BSON-IDs for its records, which give pseudo-random values which are timeseries ordered. These are great for single-table designs in serverless architectures as they can be generated in a distributed fashion, and when stored in a sort key give a natural index ordered by document creation.

Unfortunately, BSON itself is a bit painful to work with in plain JavaScript, but there are other options like KSUID which achieve much the same thing, but with better randomness and ease-of-use.

If time-series ordering is not important to you, functions like nanoid can achieve much of the same outcomes without needing to worry about hash collisions (and while being URL-friendly too).

Do some upfront planning and future thinking

It can be hard with agile-development methodologies to get into the habit of planning architecturally ahead, but single-table design with DynamoDB typically assumes you know all of your access patterns upfront in order to structure your data optimally.

In reality, you don't need to know everything upfront, but it is ideal to have some idea about how your data will be accessed into the short to medium term. The advantages of doing so include:

indexing fields before-hand - this prevents needing to back-fill later (especially while your data set is small),
pre-allocating indexes before they are used - even if you don't fill them, CloudFormation only permits deploying one new Global Secondary Index (GSI) at a time, so it can be good to look ahead and add unused indexes before you need them so you aren't coordinating multiple deploys to get them in place before you deliver code that requires them
managing document size - not only thinking about how your data is accessed, but how it will grow, can help you make earlier decisions about splitting data into multiple collections or simply storing it in your original document. While it is always easy to add new collections later, finding that your documents are growing in size as time goes on means that you still need to perform a painful data migration step when you want to refactor it

Work with the databases limitations, not against them

DynamoDB is deliberately structured with a limited set of data management features, which makes it hard to translate relational data modelling with normalisation to its model.

Learning DynamoDB means letting go of a lot of those techniques and learning new techniques to achieve the same principles you require for data integrity, performant access and ease-of-access.

For example, this means using streams to propagate changes to other records or other systems, identifying when you really need full read consistency, not being afraid to duplicate data between indexes and even other documents when it makes sense for your data patterns.

This even means realising when DynamoDB is a good fit for your use cases, and when something else may be complementary, like ElasticSearch for text based search, or exporting to S3 and scanning your data with Athena for analytics use cases.

The key part of this is learning about all its features (not just its API operations, even though you should be familiar with every part of them to use them successfully) and when to apply them.

I hope some of this has been useful, and if you have any of your own recommendations from your experience modelling on DynamoDB, I'd love to hear them. You can give me a shout-out on Twitter where I'm often posting or re-tweeting on AWS or JavaScript related things.

When to use DynamoDB

Chris Armstrong — Tue, 17 Aug 2021 05:50:00 +0000

(This was originally posted at my development blog on chrisarmstrong.dev. Check it out for more AWS and DynamoDB related articles!)

There's no doubt DynamoDB has surged in popularity hand-in-hand with the growth in serverless adoption, as well as those fascinated and horrified by Rick Houlihan's talks from AWS re:Invent explaining single-table design.

Here I run through some of the key advantages and disadvantages you might want to consider when evaluating DynamoDB for your next project.

Why use DynamoDB over a SQL database

Operating a DynamoDB database is simplified as much as possible. DynamoDB is a fully-managed service, so you are freed from the headaches of managing servers, capacity planning, scheduling and coordinating backups, multi-AZ setup and other operational concerns that come with self-managed databases.

Compared to SQL, operational overhead related to schema management is also gone, because you can declare your tables in CloudFormation or Terraform, and the (almost) schemaless document-design lets you worry about evolving your document structure in code (instead of as a generally forgotten operational concern to be addressed
at release time).

DynamoDB comes with a number of automated operational concerns, not limited to:

automatic data replication between availability zones for high data safety and replication
automatic capacity scaling with usage that is responsive and related to cost
data replication between AWS regions using global tables
quick and simple backups
point-in-time recovery
builtin encryption at rest, including with your keys

A well-thought out single-table design allows your schema to evolve with minimal fuss. Although data migrations may be required for some significant changes, adding fields and new collections to your table is simply just a matter of defining them in your code. Provided the relationships of your core document types are well understood from the beginning, you do not find yourself having to manually run Data-Definition Language (DDL) statements on deploy or put in a place a scheme
to apply them.

You get a consistent performance profile as your application grows. DynamoDB scales from hundreds of rows to millions of rows while delivering the same performance. Even so-called "small" applications have one or two tables that grow at an exponentially faster rate than others, which eventually slows down the performance of the entire application.

Provided you use DynamoDB properly, you never hit that 'we need to optimise our queries' moment, or slowdowns from large tables affecting the rest of the application.

SQL databases let developers hide performance issues from themselves, which only surface as the usage of a system starts to grow. Designing for DynamoDB requires developers and system architects to understand their access patterns upfront, which means that most potential performance issues can be addressed from the beginning.

With SQL, problems only manifest as tables get bigger and queries no longer perform as well as when the system was small and lightly used. By that stage, you are rushing to understand bottlenecks and fix them while the user experience suffers.

Understanding the performance profile of DynamoDB is easier for developers to understand compared with SQL. Because SQL
gives the illusion of being able to access your data flexibly and performantly when databases are small, the underlying complexity of indexes, joins and access plans (and the seemingly inscrutable decisions a database makes to implement them) can be a surprising and steep learning curve which becomes urgent at the worst times.

DynamoDB on the hand has a very simple (i.e. restrictive) API and set of storage mechanisms available, and therefore understanding what does and doesn't scale in DynamoDB is much easier. Add in the predictable latency of each operation at scale with a simple set of rules, and the performance profile becomes accessible to most developers.

When DynamoDB is inappropriate

DynamoDB for analytics use cases is hard. It's highlighted again and
again that DynamoDB excels at OLTP (transactional) use cases and is
inappropriate for OLAP (analytical and data warehousing) use cases which need to perform complex joins and run over large datasets in the one query. Without going any further into this, unless you understand your analytics use cases well and can build real-time analytics processing with DynamoDB streams, you should look elsewhere for your OLAP requirements.

Text search is inefficient Another common use case is free text search or cases where you need lots of different indexes - although DynamoDB allows up to 50 GSIs, this comes with a high cost and complexity. An even more awful solution would be to scan and read all the documents on demand.

As awful as it is, ElasticSearch is better start, and a hybrid-architecture can easily be achieved by using DynamoDB streams to feed updates into a paired ElasticSearch cluster.

Modelling your database upfront is difficult. Although you can get away with a small amount of upfront modelling, if your application is pivoting regularly and the business domain is really uncertain, it can be hard to work with DynamoDB when your schema is constantly shifting, doubly-so if you already have something in production and need to migrate data.

This is where something like SQL may present some advantages as your schema evolves, but you will still be fighting schema changes and performance issues caused by a lack of planning or consideration of performance in the long run.

Forrest Brazeal has put together a more comprehensive set of reasons why you shouldn't use DynamoDB, with some really important points, especially regarding developers not understanding how to use DynamoDB properly and attempting to replicate patterns from MongoDB and SQL databases and quickly running into trouble (a topic I have plenty of thoughts about and will write about soon).