DEV Community: Frank Wang

Work on your Lambda functions live

Frank Wang — Tue, 26 Jan 2021 15:32:57 +0000

AWS CDK has been great for us and our readers over on Serverless Stack. But the local development experience for Lambda isn't great. To fix this we created a Live Lambda Development environment in SST. SST makes it easier to build serverless apps by allowing you to work on your Lambda functions locally while connecting to your deployed resources.

Before we look at how it works, let's look at what the local development environment for serverless is usually like.

Background

Working on Lambda functions locally can be painful. You have to either:

Locally mock all the services that your Lambda function uses

Like API Gateway, SNS, SQS, etc. This is hard to do. If you are using a tool that mocks a specific service (like API Gateway), you won't be able to test a Lambda that's invoked by a different service (like SNS). On the other hand a service like LocalStack, that tries to mock a whole suite of services, is slow and the mocked services can be out of date.
Or, you'll need to deploy your changes to test them

Each deployment can take at least a minute. And repeatedly deploying to test a change really slows down the feedback loop.

Introducing `sst start`

To fix this we added the sst start command to the Serverless Stack Toolkit (SST). It's an extension to AWS CDK that makes it easier to build serverless apps using CDK.

This command does a couple of things:

It deploys a debug stack with a WebSocket API to the same AWS account and region as your app.
It deploys your app and replaces the Lambda functions with a stub Lambda.
Starts up a local WebSocket client to connect to the debug stack.

The debug stack contains a serverless WebSocket API and a DynamoDB table. The stub Lambda when invoked, sends a message to the WebSocket API, which in turn sends a message to the local client connected to it. The client then executes the local version of the Lambda function and sends back the results to the WebSocket API. Which then responds to the stub Lambda. And finally the stub Lambda responds back with the results.

An Example

Let's look at an example.

In this sample app we have:

An API Gateway endpoint
An SNS topic
A Lambda function (api.js) that responds to the API and sends a message to the SNS topic
A Lambda function (sns.js) that subscribes to the SNS topic

So when a request is made to the API endpoint, the stub version of api.js gets invoked and sends a message to the debug stack. This in turn gets streamed to the client. The client invokes the local version of api.js and returns the results to the debug stack. The local version also sends a message to the SNS topic. Meanwhile, the stub api.js responds to the API request with the results. Now the stub version of sns.js gets invoked as it is subscribed to the SNS topic. This gets sent to the debug stack which in turn gets streamed to the client to execute the local version of sns.js. The results of this are streamed back to stub sns.js that responds with the results.

You can try out this sample repo here.

Advantages

This approach has a couple of advantages.

You can work on your Lambda functions locally
While interacting with your entire deployed AWS infrastructure
It supports all Lambda triggers, so there's no need to mock API Gateway, SQS, SNS, etc.
It supports real Lambda environment variables
And Lambda IAM permissions, so if a Lambda fails on AWS due to the lack of IAM permissions, it would fail locally as well.
And it's fast! There's nothing to deploy when you make a change!

A couple of things to note.

The debug stack is completely serverless
- So you don't get charged when it's not in use
- And it's very cheap per request, it'll be within the free tier limits
All the data stays between your local machine and your AWS account
- There are no 3rd party services that are used
- Support for connecting to AWS resources inside VPC is coming soon

Conclusion

The sst start command in SST creates a live lambda development environment. By extending AWS CDK, it allows you to build serverless applications in CDK while having a first-class local development environment.

As one of our early users put it; sst start is the future of Lambda development

If you like the SST project, consider starring our repo —

sst / sst

SST v2

SST makes it easy to build modern full-stack applications on AWS. Watch the SST in 100 seconds video to learn more.

$ npx create-sst@latest

For v2 docs, head over to v2.sst.dev

Pick your frontend

Deploy Next.js, Svelte, Remix, Astro, Solid, or any static site to AWS.

Add any feature

SST gives you the full power of AWS. Making it easy to add any feature to your product.

File uploads — Allow your users to upload files to S3.
Auth — Authenticate your users through any auth provider.
Events — Run tasks after your app has returned to your user.
Databases — Use a serverless SQL or NoSQL database to power your app.
Jobs — Run cron jobs or long running jobs powered by serverless functions.
APIs — Add a dedicated serverless REST, GraphQL, or WebSocket API to your app.

Collaborate with your team

Finally, you…

View on GitHub

Using Serverless Framework and CDK together

Frank Wang — Wed, 16 Sep 2020 02:31:12 +0000

While helping folks work through our Serverless Stack guide, we noticed that most folks have a really hard time using CloudFormation templates to define their AWS infrastructure. And so it's no surprise that we are really excited about AWS CDK! It allows you to define any AWS infrastructure using modern programming languages. Making CDK, truly infrastructure as code.

Serverless Framework on the other hand is great for building Serverless applications on AWS. While you can build Serverless apps purely with CDK, it doesn't have a great developer experience. On the flip side, Serverless Framework allows you to deploy any non-Serverless AWS infrastructure. But it requires you to use CloudFormation templates written in YAML or JSON.

The overlap between these two frameworks have led folks to write about why you should use one versus the other. However, we think they are meant to be used together, in tandem. And we've created a tool to do so — Serverless Stack Toolkit (SST). Allowing you to get the best of both worlds!

Let's look at this in detail.

Background

Before we dive in, let's quickly get some background on these frameworks.

What is Serverless Framework

Serverless Framework was released back in October 2015. It allows you to develop Serverless applications locally and deploy them with ease. It's driven by a serverless.yml config file that looks something like this:

service: serverless-example

provider:
  name: aws
  runtime: nodejs12.x

functions:
  hello:
    handler: handler.hello

You can invoke your Lambda functions locally using:

$ serverless invoke local -f hello

And deploy them by:

$ serverless deploy

Serverless Framework will automatically package your Lambda functions and upload them to AWS.

You can also add other AWS infrastructure using CloudFormation templates in YAML. For example, you can add a DynamoDB table using:

Resources:
  NotesTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: ${self:custom.tableName}
      AttributeDefinitions:
        - AttributeName: userId
          AttributeType: S
        - AttributeName: noteId
          AttributeType: S
      KeySchema:
        - AttributeName: userId
          KeyType: HASH
        - AttributeName: noteId
          KeyType: RANGE
      BillingMode: PAY_PER_REQUEST

What is AWS CDK

AWS CDK (Cloud Development Kit) was released in Developer Preview back in August 2018. It allows you to use TypeScript, JavaScript, Java, .NET, and Python to create AWS infrastructure.

So for example, here is how a CloudFormation template for a DynamoDB table would compare to the CDK version:

- Resources:
-   NotesTable:
-     Type: AWS::DynamoDB::Table
-     Properties:
-       TableName: ${self:custom.tableName}
-       AttributeDefinitions:
-         - AttributeName: userId
-           AttributeType: S
-         - AttributeName: noteId
-           AttributeType: S
-       KeySchema:
-         - AttributeName: userId
-           KeyType: HASH
-         - AttributeName: noteId
-           KeyType: RANGE
-       BillingMode: PAY_PER_REQUEST


+ const table = new dynamodb.Table(this, "notes", {
+   partitionKey: {
+     name: 'userId', type: dynamodb.AttributeType.STRING,
+   },
+   sortKey: {
+     name: 'noteId', type: dynamodb.AttributeType.STRING,
+   },
+   billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
+ });

It also allows you to create higher order constructs. This means that you can combine and compose lower level resources to create new reusable constructs.

Similar to Serverless Framework, you can deploy a CDK app using:

$ cdk deploy

It also allows you to define Lambda functions as well.

const helloSrc = fs.readFileSync('handler.js').toString();

const helloLambda = new lambda.Function(this, 'HelloLambda', {
  runtime: lambda.Runtime.NODEJS_12_X,
  handler: 'handler.hello',
  code: new lambda.InlineCode(helloSrc, {
    encoding: 'utf-8',
  })
});

However, it doesn't offer a great local development experience for your Lambda functions.

Using Serverless Framework and CDK together

To recap, here's how Serverless Framework and CDK compare.

	Serverless Framework	AWS CDK
Perfect for Lambda functions	✅	〰️
Perfect for other infrastructure	〰️	✅

So if you are building a Serverless application that includes other AWS infrastructure, it would make a ton of sense to use Serverless Framework for your Lambda functions and CDK for the rest.

As an aside, a similar design pattern of splitting up your Lambda functions from the other infrastructure is common in the Serverless community and we recommend it over on Serverless Stack as well.

However, there are a couple of design differences between the two that you need to be wary of.

Design differences between Serverless Framework and CDK

Serverless apps are made up of multiple services and the app as a whole is deployed to the same environment.

Each service is deployed as a CloudFormation stack to the target AWS account. You can specify a stage, region, and AWS profile to customize this.

 $ AWS_PROFILE=development serverless deploy --stage dev --region us-east-1

The --stage option here prefixes your stack names with the stage name. So if you are deploying multiple stages to the same AWS account, the resource names will not thrash.

This allows you to easily deploy your Serverless app to multiple environments. Even if they are in the same AWS account.

AWS CDK apps on the other hand are made up of multiple stacks. And each stack is deployed to the target AWS account as a CloudFormation stack. However, unlike Serverless apps, each stack can be deployed to a different AWS account or region.

This means that each time you deploy your CDK app, it could potentially create a stack in multiple environments. This critical design difference prevents us from using CDK apps alongside our Serverless services.

You can fix this issue by following a certain convention in your CDK app, as outlined in this post — Deploy aws-cdk app to different environments. However, this is only effective if these conventions are enforced.

Enter, Serverless Stack Toolkit

To fix this, we created the Serverless Stack Toolkit (SST).

SST allows you to follow the same conventions as Serverless Framework. This means that you can deploy your Lambda functions using:

$ AWS_PROFILE=production serverless deploy --stage prod --region us-east-1

And use CDK for the rest of your AWS infrastructure:

$ AWS_PROFILE=production npx sst deploy --stage prod --region us-east-1

Just like Serverless Framework, the stacks in your CDK app are prefixed with the stage name. Now you can use Serverless Framework and CDK together! Allowing you to do something like this:

Note that, if you define the AWS account or region as a part of your CDK stack, like so:

new MyStack(app, "my-stack", { env: { account: "1234", region: "us-east-1" } });

You'll get an error.

Error: Do not directly set the environment for a stack

And there's more!

SST also has a couple of other key advantages.

Concurrent deployments

AWS CDK deployments are currently very slow. CDK deploys your CloudFormation stacks in sequence. It'll submit a CloudFormation template for deployment and wait till it completes, before starting the next one. This means that CDK deployments for large apps can easily take at least half an hour. SST fixes this by deploying your CloudFormation stacks concurrently.
Asynchronous deployments

SST also supports deploying your CloudFormation stacks asynchronously. So you don't have to waste CI build minutes waiting for CloudFormation to complete. Seed natively supports concurrent asynchronous deployments for your SST apps. Making it nearly 5 times faster and virtually free to deploy!
Supports ES6 (and TypeScript) out-of-the-box
Automatically lints your CDK code using ESLint

Differences between SST and CDK

SST is a simple extension of CDK with a couple of minor differences.

There is no cdk.json

Instead there is a sst.json with the defaults for your stage and region.

{
  "name": "my-sst-app",
  "type": "@serverless-stack/resources",
  "stage": "dev",
  "region": "us-east-1"
}

There is no bin/*.js

Instead there is a lib/index.js that has a default export function where you can add your stacks. Note that, SST creates the App object for you.
```
import MyStack from "./MyStack";

export default function main(app) {
  new MyStack(app, "my-stack");

  // Add more stacks
}
```

Stacks extend sst.Stack

Your stack classes extend sst.Stack instead of cdk.Stack. This allows SST to prefix your stack names with the stage you are deploying to.

import * as sst from "@serverless-stack/resources";

export default class MyStack extends sst.Stack {
  constructor(scope, id, props) {}
}

You can read more about moving a CDK app to SST here. So if you'd like to use SST but have an existing CDK app, you can do so by following a couple of easy steps.

Summary

We think AWS CDK is a vast improvement over CloudFormation templates that are written in YAML or JSON. And the combination of Serverless Framework and CDK allows you to leverage the best of both worlds. Develop and deploy Lambda functions using Serverless Framework and define the rest of your AWS infrastructure with CDK. To make sure that you can use the two together, we created the Serverless Stack Toolkit (SST). A simple extension of CDK that allows you to deploy CDK apps alongside your Serverless Framework services!

DEV Community: Frank Wang

Work on your Lambda functions live

Background

Introducing sst start

An Example

Advantages

Conclusion

sst / sst

SST v2

Pick your frontend

Add any feature

Collaborate with your team

Using Serverless Framework and CDK together

Background

What is Serverless Framework

What is AWS CDK

Using Serverless Framework and CDK together

Design differences between Serverless Framework and CDK

Enter, Serverless Stack Toolkit

And there's more!

Differences between SST and CDK

Summary

Introducing `sst start`