DEV Community: Serverless Inc.

Serverless monitoring — the good, the bad and the ugly

We're Serverless! — Fri, 03 Jun 2022 14:55:45 +0000

Originally posted at Serverless on September 26th, 2017

Not so long ago, a job requirement pushed me into the world of FaaS, and I was thrilled. I had dreams of abstraction — eliminating all that tedious work no developer likes doing. “We are not operations engineers!” I exclaimed proudly. “We should not need to dabble in the dark arts of the Linux Shell.”

But little did I know how wrong I was. We humans are creatures of habit, and one of my habits as an AWS user is checking the AWS Console religiously. It was my central place to monitor everything I needed to know about my servers’ health.

Now comes the difficult question: How does monitoring work when using AWS Lambda and Serverless?

Monitoring 101

All applications have metrics we, as developers, need to monitor. This is crucial: downtime and slow apps can create some pretty grumpy customers.

Trust me, I get angry phone calls and rage mail every once in a while. So how can you avoid getting yelled at by customers? Track your errors and monitor your software!

Implement a good notification system that lets you know when and where an error occurred. Make sure to have good and easy to view logs of all errors, warnings and other crucial data your application creates. Be responsible for the software you write. Because it is our legacy as developers. We have made an oath, to be creators of awesome stuff!

But user experience is only one side of the performance metrics. The second crucial metric is the measure of computational resources. How much resources is the app consuming. If it is too much you need to scale down your servers, otherwise, if the app is capping all resources you may consider larger servers or more of them.

**Note: I recently came across an awesome article on this topic by none other than the CTO of RisingStack, Peter Marton. He explained in detail how to do monitoring right. I urge you to take a peek, it will change your view on monitoring forever.

Overhead?

Excuse me…? Can I have some monitoring, please? But, without being a burden on my application.

We’re lucky that, in 2017, this is a given. Monitoring software has become so advanced that in today’s world of programming the overhead is minimal. The sun was not shining so bright back in the day. Monitoring applications was followed by a known fact that it would impact your app’s performance significantly.

How does this translate to Serverless?

The Serverless revolution has been gaining strength for the past few years. I see no reason for it to stop. The hype is real.

Developers are starting to view the Function as a S ervice architecture as a savior, something that makes it possible to scale applications automatically and serve only as many users as needed. The pay-as-you-go method cuts costs drastically and makes it possible for startups to create awesome software for a fraction of the cost.

But, wait a minute. What else needs to be cut for that to become a possibility?

A couple of things come to mind. The overview of your code performance and tracking errors are first. Silent failures as well. How do you monitor the performance of a server that is not a server? Schrödinger’s server? Okay, now my head hurts.

This paradox needs a new perspective. Monitoring Serverless is a new beast in itself. Traditional methods will not work. A new mindset is in order.

server != functions

Instead of telling our functions to send along additional data with every invocation, why not just collect their residual data? This is a cool idea! It’s a known fact all AWS Lambda functions send their logs to AWS CloudWatch.

Serverless is unforgiving

Unlike in traditional applications, you don’t have full overview of every part of your system. Not to mention how hard it is to test Serverless. You have to push code to AWS to see if it’s working or spend an eternity on setting up emulators on your local machine. The process is incredibly tedious. Not to start with adding third-party services to your app. It creates overhead and additional costs. Try attaching monitoring services to every single Lambda function. That’s never going to scale well!

Let’s imagine a scenario of monitoring a simple function on AWS Lambda. The purpose is to test the function and check the verbosity of the logs on CloudWatch.

After hitting the endpoint with Postman a couple of times I’m assured it works fine.

Opening up CloudWatch I can see the logs clearly. All the function invocations are listed.

The logs are extensive, the only issue is I can’t seem to make any sense of them. I can see the functions we’re invoked, but not much else. Error messages for failing functions are not verbose enough, so they often go unnoticed. I’m also having a hard time finding functions that timed out.

I also tried logging through the command line. It shows possible errors a bit better, but still, not good enough to have peace of mind.

serverless logs -f my-function

Not to mention the tiresome nature of having to push code to AWS every time you’d want to try out something new. Thankfully, all is not lost.

Making my life less miserable

What if I didn’t need to push code to AWS every time I wanted to test something? All heroes don’t wear capes. Like a knight in shining armor, Serverless Offline comes barging in to save the day! At least now I can test all my code locally before pushing it to AWS. That’s a relief.

Setting it up is surprisingly easy. Installing one npm module and adding a few lines to the serverless service’s serverless.yml and voila, API Gateway emulated locally to run Lambda functions.

Switching to the directory where I created the sample function and service, I just ran the following command in a terminal:

npm install serverless-offline --save-dev

After installing serverless offline I just referenced it in the serverless.yml configuration:

Back in my terminal running Serverless offline is as easy as just typing:

serverless offline start

That’s it, a local development simulation of API Gateway and Lambda is up and running!

The logs are still bad though…

I still can’t get over the fact how bland the logs are. Not to mention the lack of error reporting. I took me a good while to find failing functions in the logs. Imagine the nightmare of tracking them in a large scale production application. This issue is what bothers me to most. The lack of overview. It’s like swimming in the dark. I don’t have the slightest clue what’s down there.

What did I do? I went hunting. There has to be something out there on the web that can help me out. I was looking for a way to simulate the monitoring and logging of a server. I thought maybe there’s a way to create a broader perspective over the whole serverless system. What I found blew me away, in a good way. A bunch of tools exist that parse and analyze logs from all functions in a system on the account level. Now that’s cool.

I decided to try out Dashbird because it’s free and seems promising. They’re not asking for a credit card either, making it a “why not try it out” situation.

They say it only takes 5 minutes to hook up with your AWS account and be ready to go, but hey. I’m a skeptic. I timed myself.

The onboarding process was very straightforward. You just add a new policy and role on your AWS account, hook it to your Dashbird account and that’s it. They even have a great getting started tutorial.

If you want to know, the timer stopped at 4 minutes. I’m impressed.

However, I’m much more impressed with Dashbird. I can finally see what’s going on.

Errors are highlighted, and I can see the overall health of my system. I feel great all of a sudden. It also tracks the cost so I don’t blow the budget. Even function tailing in real-time is included. Now that’s just cool.

With this watching my back I’d be comfortable with using Serverless for any large-scale application. The word relief comes to mind.

Final thoughts

Whoa… This has been an emotional roller-coaster. Starting out as a skeptic about the ability to monitor and track large-scale Serverless apps, I’ve turned into a believer.

It all boils down to the developer mindset. It takes a while to switch from the mental image of a server to FaaS. Serverless is an incredible piece of technology, and I can only see a bright future if we keep pushing the borders with awesome tools like Serverless Offline, Dashbird, CloudWatch, and many others.

I urge you to check out the tools I used above, as they have been of great help to me.

Hope you guys and girls enjoyed reading this as much as I enjoyed writing it. Until next time, be curious and have fun.

Do you think this tutorial will be of help to someone? Do not hesitate to share. If you liked it, let me know in the comments below. Tools: Resources:

Originally published at https://www.serverless.com.

How To Manage Your Alexa Skills With Serverless

We're Serverless! — Thu, 02 Jun 2022 22:42:02 +0000

Originally posted at Serverless on January 8th, 2018

Masashi here, creator of the Serverless Alexa plug-in.

Serverless and IoT go hand in hand, and it’s easy to use the Serverless Framework to develop AWS Lambda functions for Alexa Skills.

Unfortunately, you can’t control Alexa Skills with the Framework, which was a bummer to me because I found the Alexa Skills Kit webapp and ask-cli didn’t have the simplicity I’d come to love with the Serverless Framework.

But! Luckily, the Serverless Framework has a great plugin system. I decided to solve this little problem with the power of the community!

The Serverless Alexa Skills Plugin lets you integrate Alexa Skills into the Serverless Framework. We can now control the manifest and interaction model of Alexa Skills using sls command and serverless.yml!

Installation

The plugin is hosted by npm:

$ npm install -g serverless
$ sls plugin install -n serverless-alexa-skills

Get your credentials

To get your credentials, log in to the Amazon Developer Console, go to Login with Amazon from APPS & SERVICES, and then Create a New Security Profile:

For following columns, you can enter whatever you like:

Go to the Web Settings of the new security profile:

Allowed Origins can be empty. Enter http://localhost:3000 in Allowed Return URLs. This port number can be changed with serverless.yml, so if you want to change this, please do so:

Remember your Client ID and Client Secret of the new security profile, as well as Vendor ID. You can check your Vendor ID at here.

You only need to do this process once. You can continue to use the same credentials as long as you use the same account.

The troublesome browser click-work is over! 👏 Let’s move on to the sls command.

Put your credentials into the Framework

Write the Client ID, Client Secret, and Vendor ID to serverless.yml. It is good to use environment variables as it is shown below.

Change the port number with Allowed Return URLs of Login with Amazon, add localServerPort, and write the port number:

Then, execute the following command:

$ sls alexa auth

This command opens the login page of Amazon.com in your browser. You will be redirected to localhost:3000 after authenticating. If the authentication is successful, you'll see the message: "Thank you for using Serverless Alexa Skills Plugin!!".

note: The security token expires in 1 hour. Therefore, if an authentication error occurs, please re-execute the command. I’m planning to implement automatic token refreshing in the future.

Let’s make a skill!

To start, execute the following command:

$ sls alexa create --name $YOUR_SKILL_NAME --locale $YOUR_SKILL_LOCALE --type $YOUR_SKILL_TYPE

These are descriptions of the options:

name: Name of the skill
locale: Locale of the skill (en-US for English, ja-JP for Japanese and so on)
type: Type of the skill (custom or smartHome or video)

A manifest is initially set for the skill. You can check the manifest with the following command:

Copy [Skill ID] and [Skill Manifest] and paste it to serverless.yml as below.

Execute the following command to update the manifest after updating your serverless.yml (or you can use the --dryRun option to check the difference between the local setting and the remote setting without updating):

$ sls alexa update

You can see the format of the manifest here.

Building the interaction model

The skill does not have an interaction model at first, so you’ll need to write an interaction model definition to serverless.yml.

Like this!

You can see the format of the interaction model here.

Execute the following command to build the model after updating your serverless.yml (and you can also use the --dryRun option with this command):

$ sls alexa build

Then, you can check the model like so:

There are a few more steps needed in order to completely publish skills, so I’m planning to do further integrations with the Alexa Skills Kit in the future. It’s still pretty great to be able to integrate manifests and models, since we update those many times as we develop. All the better if we can manage them with the source code of our Lambda functions!

Now, we can completely manage our Lambda Functions and Alexa Skills with Serverless Framework + Serverless Alexa Skills Plugin!

If you have any comments or feedback, please create an issue or send a pull request. I always welcome them 🍻

Originally published at https://www.serverless.com.

CI/CD for monorepos

We're Serverless! — Thu, 02 Jun 2022 15:59:40 +0000

Originally posted at Serverless on March 6th, 2020

This article was updated to reflect changes in the new dashboard at app.serverless.com

When building serverless applications as a collection of Serverless services, we need to decide whether we are going to push each service individually to our version control system, or bundle them all together as a single repo. This article is not going to go into the details about which is better or not, but all our posts so far seem to show examples of services all stored in individual repositories.

What this article is going to demonstrate, however, is that deploying services from within a single monorepo is easily doable within Serverless Framework Pro’s CI/CD solution.

Getting started

You need to make sure that the services you are deploying are bundles together in a repo in separate subdirectories off the root of the repo. You can see a simple example repo here to see how this is structured to make sure that yours matches as closely as possible. Really the biggest part to take note of is that all services are off the root of the repo as separate sub-directories and a f9older with shared code is also off that root. This just simplifies configuration later.

**Note: **Serverless Framework Pro has a generous free tier so you don’t need to worry about not having a paid account just to try this feature out for yourself.

Once you have a repo setup (or you’ve cloned the sample repo), make sure each service in each subdirectory have the same app and org settings to connect to the dashboard and that those changes are also pushed to the repo.

The last step before we walk through setting up the monorepo deployment is to ensure that we have our connection to our AWS account all squared away, especially if you have a brand new dashboard account. Here is a 2 minute video that shows you how to easily and quickly connect to AWS using Providers.

With that out of the way, let’s get cracking!

First deploy

The best way to get started is to just deploy first and get all your services deployed to AWS and created within the dashboard. Here are the steps to follow:

Make sure you have credentials for the CLI to communicate to your Serverless account by running serverless login in the CLI and completing the login process.
If the app property you have added to your services’ serverless.yml files does not yet exist, click the create app button and choose to add an existing Serverless Framework project
During that process you can create or choose a new Provider
Once the app is created, go back to your project on the CLI, make sure to cd into the first service and run serverless deploy — stage [stageyouwanthere]. The — stage is optional since it will always default to a value of dev unless you specify otherwise.
Repeat 4 by *cd*ing into each sub-directory and deploying each service into AWS.

Even if these services are already deployed, you can deploy again. As long as nothing but the org **and **app properties have changed, all that this new deployment does is add these services to your dashboard account.

Connecting to GitHub or BitBucket

Now that everything is added to the dashboard, lets click the menu option to the right of the one of our service names and choose settings:

In the settings menu, select CI/CD and you should see the CI/CD configuration open up. If you are doing this for the first time, you have probably not connected to GitHub or BitBucket before, so just click the connect option and follow the prompts.

Once you have completed that process, you will need to choose the repository for your monorepo from the dropdown list, and since this is a monorepo, the CI/CD settings will also ask you to choose the right base directory for this specific service. Go ahead and do that!

Setting up automated deployments

Since we have the basic connection all set up now, let’s scroll down to the branch deploys` section. This is where you can now configure which branch in your repo deploys to which stage or environment. Most repos have a main branch (or master) and this is often selected as the prod stage. You can however add a develop branch that can deploy to the dev stage as in the image:

Just add any additional branch to stage mappings you want to have. This configuration will then trigger an automated deployment as soon as any code changes are made to the branch you configure; for example if a developer creates a PR to the develop branch, one that PR is merged it will automatically trigger a deployment of your service to the dev stage if configured like the image above.

NOTE: You can use **Providers* to configure a different AWS connection for each stage and **Parameters** to pass different configuration values for each stage at deployment time as well.*

You will need to repeat the above process for each sub-directory within your monorepo.

More advanced configuration

Selective deployments

By default, if you have multiple services of a monorepo configured and you merge a change anywhere in that repo, all services will redeploy, just in case. There’s no way for the system to know if there are any dependencies between the services so it cannot assume that only that one service that had a change should be redeployed. In other words, if you had 6 services and you made a change to just one, 6 seperate redeployments will occur, just in case.

However, you can configure it differently. If you open the CI/CD settings for one of the services and scroll down to expand the build settings section, you should see numerous options to help you maximise the efficiency of your CI/CD pipeline for a monorepo.

By default, the Only trigger builds on selected file changes option is not selected and means that this service will always be redeployed with any change to the git repository, even if there were no changes to the code of this service. If you only want this service redeployed when its own code is edited, check the box and you should see something like:

Automatically, the directory of the current service is selected. From this point on, servicea will only be re-deployed when its own code is edited.

Dependency deployment

But what if you actually did have services that had dependencies on each other. So in the example with servicea, we could in fact link it to serviceb and configure it so that servicea will always be re-deployed when serviceb is also edited. Just by adding a reference to the correct service directory, I can ensure this will happen:

I could of course do this for any number of services that servicea depends on and vice versa. But what if I have some kind of shared folder that servicea uses. Because we reference a directory structure in our configuration, you can point to any path in your monorepo to be watched for changes. In the example repo, we have a directory called shared that stores a number of classes and functions (or at least it could) that are re-used by multiple services. If I change anything in shared, multiple services need to redeploy.

I can accomplish this just by adding the path to shared:

In the image above, servicea will be deployed on merges to Github if changes are detected in directories servicea, serviceb or shared. And I can configure any service with any specific arrangement of dependencies I need, providing me a ton of great flexibility to deploy what I need under the right circumstances.

Monorepo deployments are much simpler to manage using Serverless Framework Pro CI/CD. But if you do have any feedback for us or want to just share, please hop into our Slack channels or Forums and let us know. Or you can even DM me on Twitter if you have any questions.‍

Originally published at https://www.serverless.com.

Monitor and debug all serverless errors

We're Serverless! — Wed, 01 Jun 2022 17:53:22 +0000

Originally posted at Serverless on September 12th, 2019

One of the most popular features of the Serverless Framework is its ability to provide monitoring with automatic instrumentation. By signing up for a free Serverless Framework account and deploying your service, it is automatically instrumented to capture all of the data needed to provide metrics, alerts, notifications, stacktraces and more.

This is especially powerful when it comes to monitoring and debugging errors. When your code throws an error, then the Serverless Framework provides a few ways to monitor and debug those errors:

You will get a New Error Type alert for your service instance, notifying you in Slack or Email that a new error was identified.
The stack trace is captured and in the Serverless Dashboard you can see the stack trace highlighting the exact line which threw the error.
The invocations & errors chart will show you the number of times errors have occurred over a span of time.
Using the invocation explorer you can search and identify the individual invocations which got the error and dig into all the details.

No wonder this is such a popular feature.

However, until today, capturing the errors only worked for cases where the error was not caught by the code and resulted in a fatal crash of the Lambda invocation. But of course we do not want our services to return 500 in such cases, so more often than not, the errors are caught and the Lambda function returns a nicer 4XX error.

Today we are launching a new addition to the Serverless Framework to help capture errors even when they are caught by your code. So let’s look at this code first.

In the example above, our lambda function handler throws an error; however, it is also caught. It calls the captureError function provided by the Serverless Framework SDK in the context object. The function was able to proceed and return a nice friendly error to the API while still capturing the error. The documentation provides more details on using the captureError method.

Now that the error is captured by the Serverless Framework, you can use the powerful dashboard features to help monitor and debug these errors. Here are a few ways you can interact with these newly captured errors in the dashboard.

When a new error is captured which hasn’t been captured before, you will get a New Error Type alert. You can also setup notifications to get notified in Slack or email, or custom SNS Topics or API endpoints.

All errors, including fatal errors and captured errors, are available in the Invocation Explorer so you can filter for invocations containing errors.

Also in the invocation explorer you can dive into the details of the individual invocation to get the details about the error, including the stack trace which was captured by the captureError method in the code.

Lastly, in the service instance overview page you can view invocation metrics and filter the results to identify the captured errors.

If you want to improve monitoring and debugging for your Serverless Framework application, getting started with the automatic instrumentation is incredibly easy. Sign up in the dashboard and follow the instructions to start a new Serverless Framework project or incorporate the dashboard features into existing services.

Originally published at https://www.serverless.com.

Efficient APIs with GraphQL and Serverless

We're Serverless! — Wed, 01 Jun 2022 15:03:43 +0000

Originally posted at Serverless on July 23rd, 2018

GraphQL can be a tool for building enlightened APIs, but it can also be a source of mystery for developers accustomed to REST.

In this post, I’ll talk about the motivations that might lead you to choose GraphQL, and how to serve a GraphQL API that will let you really take advantage of its benefits.

Here’s what we’ll be covering:

REST API design

First, let’s talk about some situations that arise in REST APIs; this directly segues into when and why you would want to use GraphQL.

Let’s say you have a REST resource to represent the products that your business offers:

GET /products/123
{
  "id": "123",
  "name": "Widget",
  "price": "$10.00"
}

And also a resource for orders by customers:

GET /orders/445566
{
  "id": "445566",
  "customerName": "John Q. Public"
  "deliveryAddress": "1234 Elm St.",
  "productId": "123",
  "quantity": 5
}

The order refers to the product by its ID. If the client needs to display the product information in the context of the order, it makes two requests: one to get the order record, and one to get the details for the product specified on the order.

You might try to improve this API in a couple ways. One would be to offer a way to retrieve the product details directly from the order number, so that you can make the two requests in parallel.

GET /orders/445566/product
{
  "id": "123",
  "name": "Widget",
  "price": "$10.00"
}

If you consistently need all the product information with the order, you might just decide to include the product information in the order resource:

GET /orders/445566
{
  "id": "445566",
  "customerName": "John Q. Public"
  "deliveryAddress": "1234 Elm St.",
  "productId": "123",
  "productName": "Widget",
  "productPrice": "$10.00"
  "quantity": 5
}

Now you’ve solved the issue of having to make multiple requests, but you’ve polluted the order object with properties from another resource, which makes it harder to use and evolve. You can fix this by keeping all product information under a single property:

GET /orders/445566
{
  "id": "445566",
  "customerName": "John Q. Public"
  "deliveryAddress": "1234 Elm St.",
  "product": {
    "id": "123",
    "name": "Widget",
    "price": "$10.00"
  }
  "quantity": 5
}

That looks pretty good! It has the benefit that any code that was written to work with the /products response will also work with "product" property from /orders.

The drawback of including the product information on the order is how it affects the backend. This may require a more expensive multi-table query with SQL, or a second query under NoSQL, or else a denormalized table that records product information with the order. It also increases the size of the response body, which can become a real problem as the REST API gets more mature and the response includes more information for different purposes.

A slightly inelegant solution is to allow the request to flag whether it wants the product information:

GET /orders/445566?omitProduct=true
{
  "id": "445566",
  "customerName": "John Q. Public"
  "deliveryAddress": "1234 Elm St.",
  "quantity": 5
}

This is not very clean, but it does allow the backend to avoid doing the extra work when product information isn’t necessary, at the cost of increased code complexity. If you repeat this design struggle many times over the lifetime of your API, you may end up with a lot of flags for different properties.

If you reach this point in your API design, then congratulations! You have partially re-invented GraphQL.

Enter GraphQL

Let’s see how a GraphQL API answers the same questions.

You’d begin by creating types for products and orders. You don’t start the API by tying things together with foreign keys, as you did with REST. Instead, the order type contains a product field as we eventually decided on in the above example:

type Product {
  id: String!
  name: String!
  price: String!
}

type Order {
  id: String!
  customerName: String!
  deliveryAddress: String!
  product: Product!
  quantity: Int!
}

You create query fields to get orders and products:

type Query {
  product(id: String!): Product
  order(id: String!): Order
}

If the client needs to get an order and the relevant product details, you only need a single query.

Since the query contains an exact statement of all the properties that it expects, the service knows by design whether it needs to fetch product information. This allows you to write a backend that minimizes database and compute time.

For example, suppose you want to know the customer name, delivery address, order quantity, product name, and product price:

{
  order(id: "445566") {
    customerName
    deliveryAddress
    quantity
    product {
      name
      price
    }
  }
}

This query would return only the requested properties:

{
 “customerName”: “John Q. Public”,
 “deliveryAddress”: “1234 Elm St.”,
 “quantity”: 5,
 “product”: {
 “name”: “Widget”,
 “price”: “$10.00”
 }
}

Making your GraphQL endpoint serverless

There are even deeper advantages to having a serverless GraphQL endpoint, which you can read more about here.

The tl;dr is that when you use GraphQL, you are relying on only one HTTP endpoint; and when you have one HTTP endpoint to connect all your clients to your backend services, you want that endpoint to be performant, reliable, and auto-scaling.

Building a GraphQL endpoint with the Serverless Framework

So, how do we build this with the Serverless Framework?

We’re going to target AWS Lambda with Node 8 in this example, and the code should be easily adaptable to other FaaS providers. You can download the code for this example here.

Using the GraphQL reference implementation in JS, we can easily create our GraphQL schema from the type declarations.

First import the utilities we need from the graphql library:

const {
  graphql,
  buildSchema
} = require('graphql')

Now you can use GraphQL schema language to specify the schema:

const schema = buildSchema(`
  type Product {
    id: String!
    name: String!
    price: String!
  }

type Order {
    id: String!
    customerName: String!
    deliveryAddress: String!
    product: Product!
    quantity: Int!
  }

type Query {
    product(id: String!): Product
    order(id: String!): Order
  }
`)

Next, we create resolvers so that queries can access our data. This is also where we make sure the resolver isn’t doing any more work than necessary. The graphql library is very flexible. Resolvers can exist for individual fields, and a resolver can either be a constant value, a function, a promise, or an asynchronous function. Functions have access to any arguments for the field via a single object parameter.

We want the database record for the product information to be retrieved only when requested, so we make the resolver for that field a function:

const database = require('./database')
const resolvers = {
  product: ({ id }) => database.products.get(id),
  order: async ({ id }) => {
    const order = await database.orders.get(id)
    if(!order) return null

return {
      ...order,
      product: () => database.products.get(order.productId)
    }
  }
}

The methods database.products.get() and database.orders.get() are both asynchronous functions, returning promises. The resolver for product simply calls through to the database. You do not need to worry about manually removing extraneous fields, since graphql-js does that for you.

The resolver for order is more complex. It uses async/await syntax to fetch the order record before returning. This allows us to get the productId for use in the resolver for the product field. Since the resolver for the product field is a function, it won’t be invoked unless the product field is actually included in the query.

All that remains is to create a handler for Lambda. Using the newer asynchronous syntax introduced by Node 8 for Lambda, this is very simple.

module.exports.query = async (event) => {
  const result = await graphql(schema, event.body, resolvers)
  return { statusCode: 200, body: JSON.stringify(result.data, null, 2) }
}

Since all of the set-up logic for the GraphQL schema is outside of the handler, this will only be executed when Lambda needs to spin up a new instance to serve requests. To enable us to query by POST request, we have to include the following in serverless.yml:

service: my-api

provider:
  name: aws
  runtime: nodejs8.10

functions:
  hello:
    handler: handler.query
    events:
      - http:
          path: /
          method: POST

That’s it. After a quick sls deploy, we can curl our new GraphQL endpoint to test the query:

curl https://lsqgfkvs2i.execute-api.us-east-1.amazonaws.com/dev/ -d '{

  order(id: "778899") {

    customerName

    deliveryAddress

    quantity

    product {

      name

      price

    }

  }

}

'

{

  "order": {

    "customerName": "Stacey L. Civic",

    "deliveryAddress": "4321 Oak St.",

    "quantity": 32,

    "product": {

      "name": "Gadget",

      "price": "$8.50"

    }

  }

}

Conclusion

You’ve now got a working GraphQL endpoint built with Serverless that scales automatically with increased traffic!

In this example, we went with a single-Lambda approach. If you want infrastructural microservices, you can also use the flexibility of resolvers to have a primary Lambda that invokes other lambdas to resolve different query fields. If you want a more in-depth solution that uses GraphQL from top to bottom, you can use schema stitching to combine multiple GraphQL APIs into one.

Other Serverless + GraphQL resources

Originally published at https://www.serverless.com.

Challenges and patterns for building event-driven architectures

We're Serverless! — Tue, 31 May 2022 22:09:41 +0000

Originally posted at Serverless on July 19th, 2017

Challenges with the Event-Driven Architecture

In my previous post, I talked about how you can use DynamoDB Streams to power an event-driven architecture. While this architecture has a number of benefits, it also has some “gotchas” to look out for. As you go down this road, you need to be aware of a few challenges with these patterns.

Changing Event Schemas

In our user creation example from the last post, we’ve been saving the user’s first name and last name together in a single fullname field. Perhaps our developers later decide they'd rather have those as two separate fields, firstname and lastname. They update the user creation function and deploy. Everything is fine -- for them. But downstream, everything is breaking. Look at the code for the Algolia indexing function -- it implicitly assumes that the incoming Item will have a fullname field. When it goes to grab that field on a new Item, it will get a KeyError exception.

How do we handle these issues? There’s no real silver bullet, but there are a few ways to address this both from the producer and consumer sides. As a producer, focus on being a polite producer. Treat your event schemas just like you would treat your REST API responses. See if you can make your events backward-compatible, in the sense of not removing or redefining existing fields. In the example above, perhaps the new event would write firstname, lastname, and fullname. This could give your downstream consumers time to switch to the new event format. If this is impossible or infeasible, you could notify your downstream consumers. The AWS CLI has a command for listing event source mappings, which shows which Lambda functions are triggered by a given DynamoDB stream. If you're a producer that's changing your Item structure, give a heads up to the owners of consuming functions.

As a consumer of streams, focus on being a resilient consumer. Consider the assumptions you’re making in your function and how you should respond if those assumptions aren’t satisfied. We’ll discuss different failure handling strategies below, but you shouldn’t just rely on producers to handle this for you.

How to handle failure

Failure handing is a second challenge to consider with Lambda functions that are triggered by DynamoDB streams. Before we talk about this, we should dig a little deeper into how Lambda functions are invoked by DynamoDB streams.

When you create an event source mapping from a DynamoDB stream to a Lambda function, AWS has a process that occasionally polls the stream for new records. If there are new records, AWS will invoke your subscribed function with those records. The AWS process will keep track of your function’s position in the DynamoDB stream. If your Lambda function returns successfully, the process will retain that information and update your position in the stream when polling for new records. If your Lambda function does not return successfully, it will not update your position and will re-poll the stream from your previous position. There are a few key takeaways here:

You receive records in batches. There can be as few as 1 or as many as 10000 records in a batch.
You may not alter or delete a record in the stream. You may only react to the information in the record.
Each subscriber maintains its own position in the stream. Thus, a slower subscriber may be reading older records than a faster subscriber.

The batch notion is worth highlighting separately. Your Lambda function can only fail or succeed on an entire batch of records, rather than on an individual message. If you raise a failure on a batch of records because of an issue with a single record, know that you will reprocess that same batch. Take care to implement your record handler in an idempotent way — you wouldn’t want to send a user multiple “Welcome!” emails due to the failure of a different user.

With this background, let’s think about how we should address failure. Let’s start with something simple, like a Lambda function that posts data about new user signups to Slack. This isn’t a mission-critical operation, so you can afford to be more lax about errors. When processing a batch of records, you could wrap it in a simple try/catch block that catches any errors, logs them to Cloudwatch, and returns successfully. For the occasional error that happens, that user isn’t posted to Slack, but it’s not a big deal. This function will likely stay up to date with the most recent records in the DynamoDB stream because of this strategy.

With our Algolia indexing function, we want to be less cavalier. If we’re failing to index users as they sign up or modify their details, our search index will be stale and provide a poor experience for users. There are two ways you can handle this. First, you can simply raise an exception and fail loudly if any record in the batch fails. This will cause your Lambda to be invoked again with the same batch of records. If this is a transient error, such as a temporary blip in service from Algolia, this should be fixed on the next invocation of your Lambda and processing will continue as normal. It’s more complicated if this is not a transient error, such as in the previous section where the event contract was changed. In that case, your Lambda will continue to be invoked with the same batch of messages. Each failure will indicate that your position in the DynamoDB stream should not be updated, and you will be stuck at that position until you either update your code to handle the failure case or the record is purged from the stream 24 hours after it was added.

This hard error pattern can be a good one, particularly for critical applications where you don’t want to gloss over unexpected errors. You can set up a Cloudwatch Alarm to notify you if the number of errors for your function is too high over a given time period or if the Iterator Age of your DynamoDB stream is too high, indicating that you’re falling behind in processing. You can investigate the cause of the error, make the necessary fix, and redeploy your function to handle the new record schema and continue your indexing as usual.

Between the “soft failure” mode of logging and moving on, and the “hard failure” mode of stopping everything on an error, I like a third option that allows us to retain the structured record in a programmatically-accessible way, while still continuing to process events. To do this, we create an SQS queue for storing failed messages. When an unexpected exception is raised, we capture the failure message and store it in the SQS queue along with the record. An example implementation looks like:

Our main handler function is very short and simple. Each record is passed through a handle_record function, which contains our actual business logic. If any unexpected exception is raised, the record and exception are passed to a handle_failed_record function, which is shown below:

Pretty straightforward — it takes in the failed record and the exception and writes them in a message to an SQS queue.

When using this pattern, it helps to think of operating on individual records, rather than a batch of records. All of your business logic is contained on handle_record, which operates on a single record. This is useful when reprocessing failed records, as you can reuse the same logic. Imagine the unexpected error in your function was due to a bug in your logic that only affected a subset of records. You can fix the logic and redeploy, but you still need to process the records that failed in the interim. Since your handle_record function operates on a single record, you can just read records from the queue and send them through that same entry point:

This is a simple reprocessing script that you can run locally or invoke with a reprocessing Lambda. It reads messages from the SQS queue and parses out the record object, which is the same as the record input from a batch of records from the DynamoDB stream. This record is passed into the updated handle_record function and the queue message is deleted if the operation is successful. This pattern isn't perfect, but I've found it to be a nice compromise between the two extremes of the failure spectrum when processing streams with Lambda.

Concurrency Limits

Finally, let’s talk about concurrency limits with DynamoDB streams. The big benefit of streams is the independence of the consumers — the Algolia indexing operations are completely separate from the process that updates the marketing team’s CRM. The development team that manages the user search index doesn’t even need to know about the marketing team’s needs or existence, and vice versa.

However, it’s not quite accurate to say that consumers are completely independent. DynamoDB streams are similar to Kinesis streams under the hood. These streams throttle reads in two ways: throughput and read requests. For throughput, you may read 2 MB per second from a single shard. For read requests, Kinesis streams have a limit of 5 read requests per second on a single shard. For DynamoDB streams, these limits are even more strict — AWS recommends to have no more than 2 consumers reading from a DynamoDB stream shard. If you had more than 2 consumers, as in our example from Part I of this blog post, you’ll experience throttling.

To me, the read request limits are a defect of the Kinesis and DynamoDB streams. If you are hitting throughput limits on your streams, you can increase the number of shards as the MB limit is on a per-shard basis. However, there’s no similar scaling mechanism if you want to increase the number of read requests. Every consumer needs to read from every shard, so increasing the number of shards does not help you scale out consumers. The entire notion of an immutable log like Kinesis or Kafka is to allow for multiple independent consumers (check out Jay Krep’s excellent book, I Heart Logs, for a better understanding of immutable logs). With the current read request limits in Kinesis and DynamoDB streams, the number of consumers is severely constrained.

Conclusion

In this post, we discussed some implementation details and some gotchas to watch out for when using stream-based Lambda invocations. Now it’s your turn — tell us what you build with event-driven architectures!

Originally published at https://www.serverless.com.

Serverless Aurora: What it means and why it’s the future of data

We're Serverless! — Tue, 31 May 2022 21:40:20 +0000

Originally posted Serverless on Dec 4th, 2017

AWS had their annual re:Invent conference last week (missed it? Check out our full recap).

AWS Lambda started the Serverless movement by releasing Lambda at re:Invent 2014. But the Lambda releases this year were run-of-the-mill incremental improvements- higher memory limits, concurrency controls, and of course, Golang support (coming soon!).

All this to say, there was nothing game-changing in the functions-as-a-service (FaaS) world itself.

Well then. Does this mean that AWS is slowing down on serverless?

Hardly.

We saw AWS asserting that serverless is more than just functions:

TJ Holowaychuk 🇺🇦

@tjholowaychuk

serverless != functions, FaaS == functions, serverless == on-demand scaling and pricing characteristics (not limited to functions)

20:58 PM - 30 Aug 2017

For a deeper explanation of this, check out Ben Kehoe’s excellent post on The Serverless Spectrum.

In five years when we look back at re:Invent 2017, we won’t be talking about the different managed container offerings. We’ll be talking about this:

AWS re:Invent

@awsreinvent

Announcing Aurora Serverless. All the capabilities of Aurora, but pay only by the second when your database is being used #reInvent

16:48 PM - 29 Nov 2017

That’s right. Serverless Aurora.

Why is Serverless Aurora so important? We first need to understand two things: the technology-driven changes in software architectures in the cloud era, and the current state of the data layer in serverless architectures.

The Architectural Evolution

Earlier this year, Adrian Cockcroft wrote a piece on the Evolution of Business Logic from Monoliths through Microservices, to Functions that blew my mind. It showed how changes in technology are driving changes in development patterns and processes. Adrian has had a front row seat for these changes over the years from his work at eBay, Netflix, and now AWS.

A bunch of unrelated technologies combined to drive these changes. Faster networks and better serialization protocols enabled compute that was distributed rather than centralized. This enabled API-driven architecture patterns that used managed services from SaaS providers and broke monoliths into microservices.

Chef, Puppet, EC2 and Docker and eventually Lambda combined to enable and promote ephemeral compute environments that reduced time to value and increased utilization. These tools were combined with the necessary process improvements from the DevOps movement to increase velocity. We’re seeing smaller teams deliver features faster with lower costs.

These changes have been huge, but the data layer has been lagging. Adrian touched on database improvements, but they aren’t as mind-blowing. They have explicit tradeoffs of simple query patterns:

Compared to relational databases, NoSQL databases provide simple but extremely cost effective, highly available and scalable databases with very low latency.

The lagging data layer is particularly problematic in Serverless architectures.

The Problem of the Serverless Data Layer

I spoke on this problem at ServerlessConf NYC in October. In short, there are two approaches you can take with databases with serverless compute: server-full *or *serverless.

Server-full databases

A server-full approach uses instance-based solutions such as MySQL, Postgres, or MongoDB. I classify them as instance-based when you can tell me how many instances you have running and what their hostnames are.

I like Postgres + Mongo because of their popularity, which means data design patterns are well-known and language libraries are mature.

However, these instance-based solutions were designed for a pre-serverless world with long-running compute instances. This leads to the following problems:

Connection Limits

Postgres and MySQL have limits of the number of active connections (e.g. 100) you can have at any one time. This can cause problems if you get a spike in traffic which causes a large number of Lambda to fire.

Networking issues

Your database instances will often have strict firewall rules about which IP addresses can access them. This can be problematic with ephemeral compute — adding custom network interfaces will add latency to your compute’s initialization.

Provisioning issues

Serverless architectures fit well with defining Infrastructure as Code. This is harder with something like Postgres roles (users). These aren’t easily scriptable in your CloudFormation or Terraform, which spreads your configuration out across multiple tools.

Scaling issues

This is one of the most important problems. Instance-based databases aren’t designed to scale up and down quickly. If you have variable traffic during the week, you’re likely paying for the database you need at peak rather than adjusting throughout the week.

Serverless databases

In contrast to server-full, instance-based databases, there is a class of serverless databases. Serverless databases are different in that you’re usually paying for throughput rather than a particular number and size of instances.

There are a few options for serverless databases, including Firebase and FaunaDB. However, the most common of these databases is DynamoDB from AWS.

DynamoDB addresses most of the problems listed above with server-full databases. There are no connection limits, just the general throughput limits from your provisioned capacity. Further, DynamoDB is mostly easy to scale up and down with some caveats. Also, the networking and provisioning issues are mitigated as well. All access is over HTTP and authentication / authorization is done with IAM permissions. This makes it much easier to use in a world with ephemeral compute.

However, DynamoDB isn’t perfect as a database. You should really read Forrest Brazeal’s excellent piece on Why Amazon DynamoDB isn’t for everyone. In particular, the query patterns can be very difficult to get correct. DynamoDB is essentially a key-value store, when means you need to configure your data design very closely to your expected query patterns.

To me, the biggest problem is the loss of flexibility in moving from a relational database to DynamoDB. With a relational model, it’s usually easy to query the data in a new way for a new use case. There isn’t that same flexibility for DynamoDB.

Developer agility is one of the key benefits of serverless architectures. Having to migrate and rewrite data is a major blocker to this agility.

The Future of Data

Ben Kehoe

@ben11kehoe

Transition to the #cloud: treat servers like cattle, not pets. Transition to #serverless cloud architecture: treat servers like roaches

11:13 AM - 25 Mar 2016

Ben Kehoe loves to hammer the point that to be truly serverless, your compute should not exist when it’s not handling data. This hyper-ephemeral compute requires a new type of database. Highly-scalable, automation-friendly, global, with a flexible data model to boot.

Distributed databases are hard. The NoSQL movement, including the Dynamo paper that describes the principles of DynamoDB and influenced its cousins (Apache Cassandra, Riak, etc.), was a first step in the database revolution.

The second step is in motion now. AWS announced multi-master Aurora, allowing for your Aurora instances to have masters that accept writes in different Availability Zones. Similarly, they announced DynamoDB Global Tables which syncs data from DynamoDB tables across different regions (!). Writes in São Paulo will be replicated to your copies in Ohio, Dublin, and Tokyo, seamlessly. These manage the difficulty of multi-master global databases.

The next step is Serverless Aurora, due sometime in 2018. It checks all the boxes for a serverless database:

✔︎ Easy scaling.

✔︎ Pay-per-use.

✔︎ Accessible over HTTP.

✔︎ Authentication & authorization over tightly-scoped IAM roles rather than database roles.

✔︎ A flexible relational data model that most developers know.

This is a big deal.

We’ve seen the hints that Amazon recognizes the issues with existing relational solutions in the cloud-native paradigm. They’ve implemented IAM authentication for MySQL and Aurora MySQL databases already. Further, the Aurora design paper notes how they have changed the relational database for a cloud-native world.

I believe this is only the first step in Amazon’s plan to push the database further. With the rise of social networks and recommendation engines, graph databases have become more popular. Amazon’s new Neptune graph database is an foray into another data area. Graph databases are notoriously hard to shard, so it may be a while before we see a Serverless Neptune. I wouldn’t bet against it coming eventually.

re:Invent is about the future, and that’s why it’s my favorite conference of the year. When we look back on re:Invent 2017, I have a feeling the data layer improvements will be the most important of all.

Originally published at https://www.serverless.com.

Using TensorFlow and the Serverless Framework for deep learning and image recognition

We're Serverless! — Tue, 31 May 2022 17:01:02 +0000

Originally posted at Serverless on July 24th, 2018

Deep and machine learning is becoming essential for a lot of businesses, be it for internal projects or external ones.

The data-driven approach allows companies to build analytics tools based on their data, without constructing complicated deterministic algorithms. Deep learning allows them to use more raw data than a machine learning approach, making it applicable to a larger number of use cases. Also, by using pre-trained neural networks, companies can start using state of the art applications like image captioning, segmentation and text analysis-without significant investment into data science team.

But one of the main issues companies face with deep/machine learning is finding the right way to deploy these models.

I wholeheartedly recommend a serverless approach. Why? Because serverless provides a cheap, scalable and reliable architecture for deep learning models.

In this post, we’ll cover how to build your first deep learning API using the Serverless Framework, TensorFlow, AWS Lambda and API Gateway.

We will cover the following:

Using serverless for deep learning — standard ways of deploying deep learning applications, and how a serverless approach can be beneficial.
“Hello world” code — a basic Lambda function with only 4 lines of code. There is no API here, we’ll start with the simplest possible example.
Code decomposition — looking through the configuration file, and the python code for handling the model, to understand how the whole example works.
API example — get a working API for image recognition on top of our example.

If you want to skip the background about what TensorFlow is and why you’d want to use serverless for machine learning, the actual example starts here.

Why Serverless + TensorFlow?

First of all, let’s briefly cover what TensorFlow is: an open source library that allows developers to easily create, train and deploy neural networks. It’s currently the most popular framework for deep learning, and is adored by both novices and experts.

Currently, the way to deploy pre-trained TensorFlow model is to use a cluster of instances.

So to make deep learning API, we would need stack like this:

The main pain points in this infrastructure is that:

you have to manage the cluster — its size, type and logic for scaling
you have to pay for unused server power
you have to manage the container logic — logging, handling of multiple requests, etc

With AWS Lambda, we can make the stack significantly easier and use simpler architecture:

The difference in both approaches

First of all, a serverless serverless approach is very scalable. It can scale up to 10k concurrent requests without writing any additional logic. It’s perfect for handling random high loads, as it doesn’t take any additional time to scale.

Second, you don’t have to pay for unused server time. Serverless architectures have pay-as-you-go model. Meaning, if you have 25k requests per month, you will only pay for 25k requests.

And not only does it make pricing completely transparent, it’s just a lot cheaper. For the example TensorFlow model we’ll cover in this post, it costs 1$ for about 25k requests. A similar cluster would cost a lot more, and you’d only achieve pricing parity once you hit 1M requests.

Third, infrastructure itself becomes a lot easier. You don’t have to handle Docker containers, logic for multiple requests, or cluster orchestration.

Bottom line: you don’t have to hire someone to do devops for this, as any backend developer can easily handle it.

As we’ll see in a minute, deploying a serverless deep/machine learning infrastructure can be done with as little as 4 lines of code.

That said, when wouldn’t you go with a serverless approach? There are some limitations to be aware of:

Lambda has strict limits in terms of processing time and used memory, you’ll want to make sure you won’t be hitting those
As mentioned above, clusters are more cost effective after a certain number of requests. In cases where you don’t have peak loads and the number of requests is really high (I mean 10M per month high), a cluster will actually save you money.
Lambda has a small, but certain, startup time. TensorFlow also has to download the model from S3 to start up. For the example in this post, a cold execution will take 4.5 seconds and a warm execution will take 3 seconds. It may not be critical for some applications, but if you are focused on real-time execution then a cluster will be more responsive.

The basic 4 line example

Let’s get started with our serverless deep learning API!

For this example, I’m using a pretty popular application of neural networks: image recognition. Our application will take an image as input, and return a description of the object in it.

These kinds of applications are commonly used to filter visual content or classify stacks of images in certain groups. Our app will try to recognize this picture of a panda:

Note: The model and example are also available here.

We’ll use the following stack:

API Gateway for managing requests
AWS Lambda for processing
Serverless framework for handling deployment and configuration

“Hello world” code

To get started, you’ll need to have the Serverless Framework installed.

Create an empty folder and run following commands in the CLI:

You’ll receive the following response:

As you can see, our application successfully recognized this picture of a panda (0.89 score).

That’s it. You’ve just successfully deployed to AWS Lambda with TensorFlow, using the Inception-v3 model for image recognition!

Code decomposition — breaking down the model

Let’s start with serverless YAML file. Nothing uncommon here-we’re using a pretty standard deployment method:

If we will look into the index.py file itself, we will see that first we need to download the model (.pb file) to the AWS Lambda .tmp folder, and then load it via a standard TensorFlow import function.

Here are the parts you have to keep in mind if you want to put your own model, with the links straight to the full code in GitHub:

**Model download from S3:**

**Model import:**

**Getting the image:**

**Getting predictions from the model:**

Now, let’s move on, and add an API to this!

API example

The simplest way to add an API to the example is to modify the serverless YAML file:

Then, we redeploy the stack:

And receive the following response:

To test the link, we can just open it in the browser:

https://.execute-api.us-east-1.amazonaws.com/dev/handler

Or run curl:

We will receive:

Conclusion

We’ve created a TensorFlow endpoint on AWS Lambda via the Serverless Framework. Setting everything up was extremely easy, and saved us a lot of time over the more traditional approach.

By modifying the serverless YAML file, you can connect SQS and, say, create a deep learning pipeline, or even connect it to a chatbot via AWS Lex.

As a hobby, I port a lot of libraries to make the serverless friendly. You can look at them here. They all have an MIT license, so feel free to modify and use them for your project.

The libraries include the following examples:

Machine learning libraries (Scikit, LightGBM)
Computer vision libraries (Skimage, OpenCV, PIL)
OCR libraries (Tesseract)
NLP libraries (Spacy)
Web scraping libraries (Selenium, PhantomJS, lxml)
Load testing libraries (WRK, pyrestest)

I’m excited to see how others are using serverless to empower their development. Feel free to drop me a line in the comments, and happy developing!

Originally published at https://www.serverless.com.

6 Things to Know Before Migrating An Existing Service to Serverless

We're Serverless! — Tue, 31 May 2022 15:49:52 +0000

Originally posted at Serverless on August 7th, 2017

Last year, my company decided to make the plunge. We were going to go Serverless! Except…most of the resources about serverless architectures are about how to start from scratch, not how to migrate existing services over.

We spent eight months figuring it out along the way and, for all of you serverless hopefuls, we made a cheat sheet. These are the steps that worked for us:

Identify the problems
Train the existing team
Create a Proof of Concept to verify that the problem is solved
Optimize the solution to take advantage of the cloud
Automate your Continuous Integration/Continuous Deployment pipeline
Automate your testing

1. Identify the problems

What problems will serverless solve that your current solution does not? We (for example) wanted serverless to help us (1) lower operational costs, and (2) give us an easier way to replace a bunch of legacy systems with a small team.

At my company, we decided that since AWS Lambda can be used as a glue between different AWS managed services, it was the best option for us.

2. Train the team

After you have defined the problems, it is easier to define (more or less) what set of technologies can help you. Your developers are the ones who already know your business logic and service requirements better than anyone else. So instead of heavily outsourcing, I’d recommend that if no one on your existing team is familiar with serverless, it is time to train them.

We encouraged our developers to go to meetups, conferences and to spend time (even work hours) tinkering with new technologies. In addition to that, we did hire a Serverless consultant to show us how to think in an event-driven manner and make sure we were following best practices.

In my opinion, one of the reasons our project was so successful was the weekly workshop meetings we held. They lasted 3–4 hours per week, and we used that time for discussing new serverless-related topics, solving issues and implementing solutions. We all got to learn a lot from each other, and workshop was very inspiring.

After you have identified the key problems to solve, and after your team has a better understanding of the tools available, it’s time to create Proof of Concept. By now, your team should have an idea of what to do next-a “hypothesis”. A Proof of Concept will help the team validate their hypothesis.

Remember: a Proof of Concept is not production code. Use it to focus on solving your problem and get rid of it. Your Proof of Concept should go to the trash after validating your hypothesis.

In our project, we developed five Proofs of Concept. Recall that our problems were: (1) to replace legacy systems and (2) reduce operational costs. Our hypotheses were as follows:

To replace legacy systems:

AWS Cognito will replace the existing authentication system
DynamoDB will replace the existing Riak database as our NoSQL database
AWS Lambda + S3 + Elastic Transcoding will replace our existing transcoding process
AWS Lambda + API Gateway + S3 will replace our existing image resizing and provide better caching mechanisms

To reduce high operational costs:

By implementing these Proofs of Concept, my team ended up validating all of our hypotheses.

4. Optimize the solution to take advantage of the cloud

This step is critically in line with the previous step. When thinking about your Proofs of Concept and new architecture, it’s important to take full advantage of the cloud. As in: don’t just grab your instances and decompose them into AWS Lambdas and API Gateways using a DynamoBD; try to think about how to take advantage of cloud-managed services, like queues and caches.

Also remember that by migrating everything to Serverless, you are transforming the architecture of your system into an event-driven architecture. In an event-driven architecture, events move around your system and all the services are decoupled from each other. AWS has a lot of services that can be used to manage events communication, like queues and streams. S3 is a great place to store your events. DynamoDB Streams can be used to let other services know that there was a change in your DynamoDB database.

5. Automate your Continuous Integration/Continuous Deployment pipeline

Since people can disagree on what exactly a ‘microservice’ is, I call a microservice a collection of AWS Lambdas and other resources related to some very specific domain, like authentication, application management or transcoding. Serverless architectures involve lots of resources being deployed into different environments in this way. When you have several moving parts, one way to make things simpler is to automate everything you can.

On my team, we used Serverless Framework to organize our projects and to automate microservice deployment into different environments. We wanted to define all our infrastructure configuration as code. The Serverless Framework helped us to do that, as all the resources needed in the microservice can be defined using Cloud Formation notation in the Serverless YAML file.

We used Jenkins Continuous Integration server to take care of running the Serverless Framework deployment in three different environments (development, staging, and production). For each environment, we used a different AWS account. We wanted to have three accounts so we could take advantage of managed resources soft limits as much as possible, and also to keep the different environments isolated.

6. Automate your testing

Testing is undervalued by a lot of teams. In a Serverless and event-driven architecture, the complexity of the code moves to the architecture itself. Because of this, testing at all levels is critical for peace of mind. Test, test, test. Also, test.

Unit tests will help you to make sure that your business logic works. For your managed services, you should write integration tests against them. It’s important to define clear interfaces between your services and the managed services so that you can have control of what is going in and out of your service.

Don’t forget about End-to-end Testing. In an event-driven architecture, the services are decoupled from each other; as a result, it can be hard to know how the events are moving around the architecture.

Conclusion

Migrating an existing service to Serverless was work, true-but honestly, it was also fun. The two most important things to have in mind when migrating one (or all) of your existing services to serverless are:

optimize your architecture to take advantage of the cloud
remember that your new architecture is an event-driven one

Best of luck, and see you on the other side.

Originally published at https://www.serverless.com.

The State of Serverless Multi-cloud

We're Serverless! — Fri, 27 May 2022 17:29:17 +0000

Originally posted at Serverless on November 6th, 2017

To multi-cloud, or not to multi-cloud

Vendor lock-in runs deep in serverless applications. “Cloud provider” used to mean “whoever hosts your servers”. In a serverless paradigm, it means “whoever runs your functions”.

And when the space doesn’t (yet) have standardization, developers must twirl those functions round and round in a whole vendor ecosystem of events and data storage. There’s no way to use Azure Functions and EC2 together.

“But,” you say, “ vendor-agnostic frameworks let you easily deploy functions across providers, at least.” That they do! But then, there’s the small technicality of language choice. Write your application in Python and you’ll have a hard time moving that over to Google Cloud Functions.

Given all this, what do we make of the multi-cloud? Is it a pipe dream, or an attainable goal-and do we actually need it?

Multi-cloud gives you wings

The biggest advantage to serverless multi-cloud we hear in the field is feature arbitrage. Imagine plucking all your favorite aspects of each cloud provider and placing them nicely together in your very own, custom-made bouquet.

It’s hard to commit to a single ecosystem, especially when serverless compute vendors are constantly adding new features that change the value equation. AWS Lambda is adding traffic shifting in Lambda aliases any day now; Microsoft Azure has their (still unique) Logic Apps, which lets you manage event-driven services much like you’re composing an IFTTT.

Pricing works out differently across vendors for different services. The same project could work better elsewhere in less than 6 months because of all this rapid feature launching. We fear lock-in because it removes our flexibility of choice.

And then, add failover into the mix.

With serverless compute, you don’t have to worry about redundancy quite as much-Lambda, for instance, automatically scales across multiple availability zones for you. But entire regions can (and do) go down.

While it’s a rare corner case, cloud outages can be devastating; we see larger companies caring more about this and moving to incorporate strategies for full cloud redundancy.

Multi-cloud gives you pause

As fun as dreamspace is, we do still have to wake up in the morning and ask ourselves: is a multi-cloud attainable and worth it?

Let’s say you want to actually try and instrument full cloud failover. The first thing you’ll have to do is write everything in the only language all four major cloud providers support. Aka: JavaScript.

Then, you’ll need to abandon your cloud databases for something like MySQL. You’ll need to constantly replicate that data from one cloud provider to another so that everything is up to date when failover occurs. You have to think, hard, about how each cloud handles logging? Secrets? Metrics?

The rule of the game is to make everything as generic as possible-which seems to go against the serverless ethos, in a way, and prevents you from utilizing those powerful features you were trying to get with multi-cloud in the first place.

It’s also worth mentioning that, for those who do choose to run an ecosystem across multiple providers, you’re paying for transfer. Not cheap.

Maybe the answer ends up being: yes, it would be cool to leverage any service I want, whenever I want, and still maintain that serverless flexibility, but things just aren’t there yet. We don’t have a Schroedinger’s cake.

The multi-way forward

There are a series of things that could happen to make multi-cloud easier.

Cross-cloud service compatibility Data management and storage, for instance, are ecosystem-dependent. Google has best machine learning right now; and while it’s feasible to use GC services on different cloud providers, it isn’t necessarily simple.

To make multi-cloud less work and less compromise, we need to be better ways to share data across cloud providers, and better ways react to any event source regardless of cloud provider.

Add shims for polyglot language support That way, it wouldn’t matter whether or not you wrote your functions in Go. Doing this yourself could be cumbersome, but soon there will probably be tools that facilitate this for you.

Smartly route your data This one’s on you, the spritely developer. Divide your application into two conceptual parts: ‘critical path’ and ‘specialized features that don’t need need to work 100% of the time’.

Anything in the critical path (things that serve your site, for instance) should be written in a cloud-agnostic way. That makes it easier to implement failover or port things over to another provider, should you need. Specialized services (e.g., image tagging) can be maintained separately for a time, or be made to process important data later in case of an outage.

The…multi-verse?

Multi-cloud probably won’t ever be completely work-free, but we expect it’ll be easy enough-sooner more than later.

And then we’ll start to see the landscape shift. Cloud providers won’t be fighting for bigger chunks of your server space; they’ll be fighting for bigger chunks of your application, in the form of features and services.

This is frankly already happening. We stereotype giants like Microsoft and Amazon as being slow to innovate, yet they’ve been rushing to push feature after serverless feature for the past two years. They’re moving faster than most startups.

As an industry, we’re headed for a user-centric software reality. Businesses will increasingly differentiate themselves with highly customized software, and multi-cloud is how they’ll do it.

Originally published at https://www.serverless.com.](url)

Deploying PyTorch Model as a Serverless Service

We're Serverless! — Thu, 26 May 2022 17:28:28 +0000

Posted at Serverless by Anand Menon

Due to the latest advancements in the Deep Learning ecosystem like improved frameworks, production ready architectures, pre-trained models …etc. building a decent model is easy (not really 😅), but the biggest question that arises afterwards is “I have built a model, what’s next ?”

A model is only as good as what use it can provide to the customers, so in order to make a model useful it should be served to millions of users in a very cost effective way. Now how do we serve or deploy a model to users? Easy, we could get on demand data, storage and computing power by leveraging any of the common cloud platforms like AWS, GCP, Azure …etc. For this tutorial we are going with AWS cloud platform.

Best things in life come for free, but not AWS resources. Cloud pricing even though very competitive can tend to stand as a hurdle for engineers to build out scalable and resource intensive products. Setting up dedicated instance infrastructure when building your MVP with AI capabilities is a suicide mission, because we have no idea about user retention, product acceptance in the market, revenue generation from the product, etc. Building up this stack using dedicated cloud infrastructure from scratch is an expensive task due to several reasons, which we will be discussing soon*.*

So a typical deep learning API stack would look as follows:

‍

As we can see this is a very complex stack and the drawback of such infrastructure is that:

We have to manage the cluster — its size, type and logic for scaling
Client has to pay for idle server power
We have to manage the container logic — logging, handling of multiple requests, etc
Requires a lot of expertise in Cloud Architecture

To solve the cons of a dedicated cloud infrastructure, cloud providers came up with serverless services (eg: AWS Lambda) whose main attractions are that, *we don’t have to manage any servers and we are billed on the number of function execution rather than on an hourly basis *(1M free requests per month).

Due to the latest advancement in the serverless ecosystem like container support, memory improvement, etc, this has opened up a lot of opportunities for all the Deep Learning practitioners to deploy models as an inference API using Lambda stack.

So today we will be deploying a PyTorch model as a Serverless API leveraging Lambda, ECR and Serverless framework. So if you want to jump right into code please check out my GitHub repo

In this tutorial, we will be deploying a simple text classification model using BERT🤗 which classifies daily user transaction logs to classes like ‘food’, ‘transport’, ‘bills’..etc and serves it as an API. I will be covering topics in detail as follows:

A brief explanation about all the resources being used
Building our model inference pipeline
Creating a Lambda function using serverless framework
Combining our inference pipeline with the lambda function
Build a docker image and testing our API locally
Tagging and deploying images to AWS ECR
Deploying lambda functions using the image deployed in AWS ECR
Finally, model inference using the serverless API

AWS Lambda Service — “With great power comes less responsibility”
AWS Lambda is basically a service which lets you run functions on cloud servers without actually managing any servers. Managing servers is never an easy task as mentioned earlier. With serverless we don’t have to think about scalability and robustness of our infrastructure, since AWS takes care of it for us.
To communicate with AWS resources like ECR, S3 ..etc programmatically we need to install AWS CLI
**Serverless Framework
**Serverless framework lets you quickly construct and deploy serverless applications using services like AWS Lambda, S3, Amazon API Gateway etc. This framework leverages AWS CloudFormation to fire up all the resources required to build our inference API using a YAML configuration file.
To install serverless framework please follow along the instructions and make sure to configure serverless with your AWS secret access keys following the guide.
**AWS ECR — **Docker 🐳 is all you need
Amazon Elastic Container Registry (ECR) is a fully managed container registry that makes it easy to store, manage, share, and deploy your container images and artifacts anywhere. So we basically build a docker image of our classifier pipeline and store it in AWS ECR.

Our complete API architecture is as shown above, here a user makes an API request with one of his daily transaction logs and this log gets passed through AWS API gateway, this request would then fire up the Lambda Function. For our initial request the lambda starts a 10GB pod and fetches the docker image from ECR to start our classifier container. The docker image comprises of a model and inference script (saving themodel in object storage is a better approach but for now we can go with this approach for simplicity). So based on the user query the lambda function performs model inference and returns a final transaction class as shown below:

Since I have explained the whole process, now we can get our hands dirty with code . I won’t be explaining about the whole BERT classifier model training pipeline, cause that is not the purpose of this blog. You can checkout my **colab notebook**to train user log classification model . After the training process is complete you will get a **pytorch_model.bin **file which we will be using as our model for building our serverless API.

Now we are going to create a python lambda function using serverless CLI command

serverless create --template aws-python3 --path serverless-logbert

The above command will create a simple boilerplate with a basic python handler script, serverless.yml, requirements.txt ..etc. Since we are building a Deep Learning text classification model using pytorch framework we need some packages that needs to be installed, so let’s add them to our requirements.txt. Since we are not leveraging GPU for inference we could go with a minimalist pytorch cpu version to save up storage.

-f [https://download.pytorch.org/whl/torch_stable.html](https://download.pytorch.org/whl/torch_stable.html)
torch==1.5.0+cpu
tqdm==4.60.0
sentencepiece==0.1.85
transformers==3.4.0

Now let’s jump directly into our handler function, The Lambda function handler is the method in your function code that processes events. When your function is invoked, Lambda runs the handler method. When the handler exits or returns a response, it becomes available to handle another event. Our handler code is as follows:

In the above code the sentence_prediction() method takes in the user input, preprocess, tokenize and pass to the trained BERT model, which in turn returns the final prediction. Currently the function returns the prediction class with highest confidence score. You can checkout the inference code here

Now we are all set to test out our inference API locally using docker. Make sure docker is installed on your local machine to test the API, please checkout the docker installation guide. Dockerfile is as follows:

FROM public.ecr.aws/lambda/python:3.8

# copy function code and models into /var/task
COPY ./ ${LAMBDA_TASK_ROOT}/

# install our dependencies
RUN python3 -m pip install -r requirements.txt --target ${LAMBDA_TASK_ROOT}

# Set the CMD to your handler 
CMD [ "handler.predict"]

Now let’s build our docker image and run our container for testing

docker build -t logbert-lambda .
docker run -p 8080:8080 logbert-lambda‍

We are now ready to test out our API locally.

The URL endpoint should be of the following format. {hostname}/{lambda-api-version}/functions/function/invocations

If it’s working in docker then it should be working everywhere else,so most of our work is done. In order for the Lambda function to fetch this image it should be deployed to AWS ECR(Elastic container registry). As the first step we need to create a repo to save our docker image, this can be done programmatically using AWS CLI as follows:

aws ecr create-repository --repository-name logbert-lambda

In order to push our image, we first need to login to ECR from our machine and this requires some identifiers like AWS region and AWS account id which we can get from AWS IAM.

We can now login to ECR using the following command:

aws_region=ap-south-1
aws_account_id=<12 digit id>aws ecr get_login-password \
--region $aws_region \
| docker login \
--username AWS \
--password-stdin $aws_account_id.dkr.ecr.$aws_region.amazonaws.com

Before pushing our image to ECR, we need to keep in mind that by default all docker images are pushed to Docker Hub, but here we need to push it to AWS ECR for lambda function to fetch our image. For that we need to tag or rename it to a format so that it will be pushed to its respective ECR repo. The format for this is as follows:

{AccountID}.dkr.ecr.{region}.amazonaws.com/{repository-name}

docker tag logbert-lambda $aws_account_id.dkr.ecr.$aws_region.amazonaws.com/logbert-lambda

Let’s check our docker image list using “ *docker image ls” *command, we will be able to see a docker image with the above format tag. Now we are all set to push our image to ECR.

docker push $aws_account_id.dkr.ecr.$aws_region.amazonaws.com/logbert-lambda

We have reached the final stage of our tutorial, that is to deploy AWS Lambda using our custom image. Now we have to edit our serverless.yml file, which was created as a boilerplate file when we created our lambda function. The following yml file let’s you configure the AWS resources that needs to be fired up when deploying our lambda function.

‍ECR makes our life super easy here, since we only need to pass the url path and digest path so that lambda could pull our locally tested image when starting up the service. We can get the URL path by either using AWS CLI or we can directly copy it from ECR console, digest can be found inside the newly created repo. Make sure to replace image PATH with our own respective URL path and digest.

Now we are all set to deploy our lambda function using the following command:

serverless deploy

The above command would fire up all the resources like AWS API gateway, lambda function, s3 bucket ..etc using AWS CloudFormation which are required for the API to function. Once the deployment process is completed we will get some logs as shown below

We are almost done 😁 now let’s do the fun part. Yes, to test our newly built API. Let’s again go back to Postman and use the URL that we got from the above serverless deployment log and test it out.

Yes!, it worked as expected and took only half a second to fetch the response, that too with CPU inference.

This serverless API infrastructure has its fair share of pros and cons, **the biggest perk being that It will automatically scale up to thousands of parallel requests without any issues. **So we don’t have to worry about building a scalable and robust architecture on our own (Means no one is gonna call you out in the middle of the night to fix up server overloads 😴 🤯).

At the same time it’s not very suitable for building production ready mission critical API due to cold start problem, but this can rectified to some extend by using AWS CloudWatch to keep our lambda service warm. GPUs are currently not available for AWS lambda which is a big disappointment 😞 for all Deep Learning folks, we can hope to see such features in the future iteration.

The future looks bright for serverless infrastructure when it comes to building AI based MVP (Minimum Viable Products) in a very cost effective way.

I hope you guys find this post useful. Always open for suggestions and criticisms. Thanks 😁

Originally published at https://www.serverless.com.

How to monitor AWS account activity with Cloudtrail, Cloudwatch Events and Serverless

We're Serverless! — Thu, 26 May 2022 15:04:27 +0000

Originally posted at Serverless on January 15th, 2018

CloudTrail and CloudWatch Events are two powerful services from AWS that allow you to monitor and react to activity in your account-including changes in resources or attempted API calls.

This can be useful for audit logging or real-time notifications of suspicious or undesirable activity.

In this tutorial, we’ll set up two examples to work with CloudWatch Events and CloudTrail. The first will use standard CloudWatch Events to watch for changes in Parameter Store (SSM) and send notifications to a Slack channel. The second will use custom CloudWatch Events via CloudTrail to monitor for actions to create DynamoDB tables and send notifications.

Setting up

Before we begin, you’ll need the Serverless Framework installed with an AWS account set up.

The examples below will be in Python, but the logic is pretty straightforward. You can rewrite in any language you prefer.

If you want to trigger on custom events using CloudTrail, you’ll need to set up a CloudTrail. In the AWS console, navigate to the CloudTrail service.

Click “Create trail” and configure a trail for “write-only” management events:

Have your trail write to a Cloudwatch Logs log group so you can subscribe to notifications:

Both examples above post notifications to Slack via the Incoming Webhook app. You’ll need to set up an Incoming Webhook app if you want this to work.

First, create or navigate to the Slack channel where you want to post messages. Click “Add an app”:

In the app search page, search for “Incoming Webhook” and choose to add one. Make sure it’s the room you want.

After you click “Add Incoming Webhooks Integration”, it will show your Webhook URL. This is what you will use in your serverless.yml files for the SLACK_URL variable.

If you want to, you can customize the name and icon of your webhook to make the messages look nicer. Below, I’ve used the “rotating-light” emoji and named my webhook “AWS Alerts”:

With that all set up, let’s build our first integration!

Monitoring Parameter Store Changes

The first example we’ll do will post notifications of from AWS Parameter Store into our Slack channel. Big shout-out to Eric Hammond for inspiring this idea; he’s an AWS expert and a great follow on Twitter:

*#awswishlist Ability to trigger AWS Lambda function when an SSM Parameter Store value changes.*
That could then run CloudFormation update for stacks that use the parameter
- Eric Hammond (@esh) December 29, 2017

Parameter Store (also called SSM, for Simple Systems Manager) is a way to centrally store configuration, such as API keys, resource identifiers, or other config.

(Check out our previous post on using Parameter Store in your Serverless applications.)

SSM integrates directly with CloudWatch Events to expose certain events when they occur. You can see the full list of CloudWatch Events here. In this example, we are interested in the SSM Parameter Store Change event, which is fired whenever an SSM parameter is changed.

CloudWatch Event subscriptions work by providing a filter pattern to match certain events. If the pattern matches, your subscription will send the matched event to your target.

In this case, our target will be a Lambda function.

Here’s an example SSM Parameter Store Event:

We need to specify which elements of the Event are important to match for our subscription.

There are two elements important here. First, we want the source to equal aws.ssm. Second, we want the detail-type to equal Parameter Store Change. This is narrow enough to exclude events we don't care about, while still capturing all of the events by not specifying filters on the other fields.

The Serverless Framework makes it really easy to subscribe to CloudWatch Events. For the function we want to trigger, we create a cloudWatchEvent event type with a mapping of our filter requirements.

Here’s an example of our serverless.yml:

Notice that the functions block includes our filter from above. There are two other items to note:

We injected our Slack webhook URL into our environment as SLACK_URL. Make sure you update this with your actual webhook URL if you're following along.
We added an IAM statement that gives us access to run the DescribeParameters command in SSM. This will let us enrich the changed parameter event by showing what version of the parameter we’re on and who changed it mostly recently. It does not provide permissions to read the parameter value, so it’s safe to give access to parameters with sensitive keys.

Our serverless.yml says that our function is defined in a handler.py module with a function name of parameter. Let's implement that now.

Put this into your handler.py file:

This function takes the incoming event and assembles it into a format expected by Slack for its webhook. Then, it posts the message to Slack.

Let’s deploy our service:

Then, let’s alter a parameter in SSM to trigger the event:

Note: Make sure you’re running the put-parameter command in the same region that your service is deployed in.

After a few minutes, you should get a notification in your Slack channel:

💥 Awesome!

Monitoring new DynamoDB tables with CloudTrail

In the previous example, we subscribed to SSM Parameter Store events. These events are already provided directly by CloudWatch Events.

However, not all AWS API events are provided by CloudWatch Events. To get access to a broader range of AWS events, we can use CloudTrail.

Before you can use CloudTrail events in CloudWatch Event subscriptions, you’ll need to set up CloudTrail to write a CloudWatch log group. If you need help with this, it’s covered above in the setting up section.

Once you’re set up, you can see the huge list of events supported by CloudTrail event history.

Generally, an event will be supported if it meets both of the following requirements:

It is a state-changing event, rather than a read-only event. Think CreateTable or DeleteTable for DynamoDB, but not DescribeTable.
It is a management-level event, rather than a data-level event. For S3, this means CreateBucket or PutBucketPolicy but not PutObject or DeleteObject.

Note: You can enable data-level events for S3 and Lambda in your CloudTrail configuration if desired. This will trigger many more events, so use carefully.

When configuring a CloudWatch Events subscription for an AWS API call, your pattern will always look something like this:

There will be a source key that will match the particular AWS service you're tracking. The detail-type will be AWS API Call via CloudTrail. Finally, there will be an array of eventName in the detail key that lists 1 or more event names you want to match.

Pro-tip: Use the CloudWatch Rules console to help configure your items the first few times. You can point and click different options and it will show the subscription pattern:

Let’s insert our DynamoDB CreateTable pattern into our serverless.yml:

Very similar to the previous example-we’re setting up our CloudWatch Event subscription and passing in our Slack webhook URL to be used by our function.

Then, implement our function logic in handler.py:

Again, pretty similar to the last example-we’re taking the event, assembling it into a format for Slack messages, then posting to Slack.

Let’s deploy this one:

And then trigger an event by creating a DynamoDB table via the AWS CLI:

Wait a few moments, and you should get a notification in Slack:

Aw yeah.

You could implement some really cool functionality around this, including calculating and displaying the monthly price of the table based on the provisioned throughput, or making sure all infrastructure provisioning is handled through a particular IAM user (e.g. the credentials used with your CI/CD workflows).

Also, make sure you delete the table so you don’t get charged for it:

Conclusion

There’s a ton of potential for CloudWatch Events, from triggering notifications on suspicious events to performing maintenance work when a new resource is created.

In a future post, I’d like to explore saving all of this CloudTrail events to S3 to allow for efficient querying on historical data-”Who spun up EC2 instance i-afkj49812jfk?” or “Who allowed 0.0.0.0/0 ingress in our database security group?”

If you use this tutorial to do something cool, drop it in the comments!

Originally published at https://www.serverless.com.